In 1985 I started a ragged little newsletter titled Denver PC Boardwatch in Denver Colorado listing 218 electronic bulletin board systems. Over the next five years it kind of grew to encompass all that was going on online including online services such as CompuServe, GEnie, and what would become America Online. In 1987 we did a story on TCP/IP networking that really didn’t have much of a name. Most just called it the Network or the Internet – a network of networks.
11 years later I sold a four-color 168-page monthly magazine, quarterly directory, and two trade shows a year to Meckler Media, and then Meckler Media to Penton Media, all in a six month period and found myself entirely cashed out at $38 million.
I was also more or less unemployed. I hung out for a few months, but no one really wanted to hear how I thought it ought to be done any more. So we parted ways mostly amicably. That they slowly tortured and then killed what I had spent 13 years building doesn’t bother me much – any longer.
And so from 1999 to 2008 I was mostly the world’s worst retired person. Me and a set of golf clubs at some of the world’s most exotic resorts is just not something anyone should have to see live. Kind of like a pig with lipstick on a set of roller blades.
Fortunately some interesting batteries led to an even more interesting electric drive 1957 Porsche Speedster replica and here we are – living retirement-free one day at a time.
Along the way, Denver PC Boardwatch morphed into Boardwatch Magazine and actually went through several “changes” of life. You see when we started covering the Internet, it was an impossibly small niche within a larger online community. In 1992 we started to tighten focus on just the Internet. There was great angst in the BBS community and among the commercial online service enthusiasts that we had somehow “abandoned them”. It was actually pretty gradual. We continued covering both to the degree they remained interesting. But Boardwatch soon became entirely about the Internet.
But the Internet grew and pretty soon there was Wired Magazine and Internet World and most computer magazines began covering the Internet. In 1994 Mosaic was released and we did the first story on this new “graphical” web browser along with the definitive tutorials on how to install Trumpet Winsock on your Windoze machine and get Mosaic up and running so you too could “see” Tim Berner-Lee’s “world wide web” of hyperlinks. But very quickly the body politic went to the matter of purchasing wine online and how to find a burp gun for your cat on the Internet. The Internet became very cool.
So we dramatically tightened our focus not just on the Internet, but specifically on the technical heart of the Internet – the large backbone networks and some 7500 “Internet Service Providers” who provided Internet service to the public. We did very technical articles on wireless networking, backbone architectures, routing tricks, and about the onslought of ever newer ever more magical hardware and software allowing faster Internet.
In doing so we became the darling of the netheads in the back room – the guys actually building out the network and frankly that is just exactly where I wanted to be.
In 1997 we developed a very unusual technique for measuring Internet connectivity at a time when it was considered entirely homogenous. To say it was controversial is an understatement. We published our first findings and subsequently endured six months of anger from almost every corner of the network, including multiple and credible death threats. In showing that not all Internet access was created equal, and actually graphing all the major backbones by performance along a series of interesting criteria, we managed to alienate almost every single player in the Internet space at the highest levels of management and of course this put enormous pressure on essentially every single network architect in the space.
It rather blindsided most of them, which was not my intent but could scarce be avoided. There were myriad lofty documents by some of the top PHd’s in networking showing in excruciating detail how terribly and deeply flawed my methodology was and an earnest effort at a coalition to declare the entire measurement theory as bogus and me personally as anathema. There was even talk of having Boardwatch “disconnected” from the Internet. That proved kind of hard to do. The Internet technically sees censorship as damage and routes around it.
The break came with enormous irony. AT&T called several times, first enraged by the poor performance depicted in the story. But eventually we got a call inquiring more collegially into specifically how we performed these measurements. I entertained it and was surprised to find the person I was talking with not only exhibited no ire, but an apparently genuine technical curiosity. Finally they revealed that they had found a router in Chicago that had had a serious programming flaw dating apparently to installation. They wondered if we could help them detect if that was the cause of the performance hit by more accurately narrowing the measurement range.
I acknowledge having all the data to do that, and entered a side bet that I could detect the exact moment they fixed the flow within a one-day time frame given the month they attempted the fix. And the bet was that I could publish the results and they were to back me up on it if I could. If I lied, they would swear to it. They agreed.
We were actually surprised at the results ourselves. We could see an enormous jump in connectivity the MINUTE they minute they pressed the ENTER key. I graphed it rather luridly. We published the before and after performance graphs and good as their word they actually issued a press release acknowledging the routing error and our detection of it. AT&T was unquestioned in the heirarchy.
At that point, it was over. We had indeed put a meter on the Internet and you could use it to detect and repair network faults. It had never been done before. We had measured the immeasurable – and all the theory of why it wouldn’t work by the very august chief priests and pharisees just didn’t matter at all. They trailed off to mumbling and muttering and eventually silence. And they didn’t MISS an issue of Boardwatch after that. It became the holy script. And if your network didn’t show up in the top 10% of backbones, there would be hell to pay.
The company we used to actually make the measurements, which involved about 10 million date points per month, was called Keynote Systems and they went on to do an IPO and provide measurement services to essentially every signficant backbone in North America. I owned a small piece of the company early on and so did rather well post IPO, but eventually disposed of it all and I’m not sure whatever became of them.
This was kind of a big deal all around. At the time, there were some enormous issues revolving around “peering” – who freely exchanged traffic with whom, where, and why. A company called Savvis Communications had entered the fray with what was dismissively termed “checkbook” backboning by actually becoming a customer of MCI, UUNet, Sprint, and several others. You HAVE to exchange traffic with your customers of course. But this tiny group in St. Louis went straight to the top of the charts on performance, and subsequently used this to do an IPO and become a successful backbone provider. But it also established that the MORE you interconnect with other networks, the better your Internet service was. The measurement series kind of ended the debate. In a network of networks, the more networks you are connected to, the better your network – DUH!
All of that to explain the always painful decision to do the same thing with EVTV. It is one of those times to tighten the focus of our publication and basically “fire” some of our admirers.
How do I know it’s time? With some melancholy. George Hamstra of Netgain Motors has kind of relocated north to better fishing and pretty much left day to day operations of Netgain Motors to his son Hunter. And most of the business has gone OEM specialty with clients such as Zamboni – the ice grooming machine.
Sebastian Bougois has closed the doors on long time converter and parts house REBIRTH AUTO. The EVnetics sister company DOES continue in business – mostly doing huge PWM controllers for railroad locomotive applications – though they will still build you a Soliton1 if you need one. He’s focusing mostly on his solar installation business these days.
A few years ago we sent our intern Matt Hauber west to find his dreams in San Diego with $1000 we gave him for a Siemens motor we had no actual use for – or even a pinout for though he PROMISED to get us one (still waiting Matt). He quickly found financial backing from Michael Bream after converting Michael’s pickup truck – founding EV West. Why is it EV West? It’s west of EVTV.
We learned this week that after pouring his soul into this startup day and night, young Hauber has left EVWest and gone out on his own doing conversions. A conversation with him yesterday indicates he couldn’t be happier. He is doing VW conversions and was working on an H2 Hummer hybrid project that could wind up being a huge deal. He has a new finance he rather fancies (actually he’s gone goofy over this lady), bought a condo in Oceanside, and life is good.
While that rather guts the technical expertise at EVWest, Bream and a new investor, Hutchison, have teamed with an early EVTV contributor Jesu Garcia to do a brand new EV show titled, comically enough, the EVShow. It’s young. It’s Hip. It’s got good music. And it’s very SoCal. And apparently about young guys in Southern California with cool electric cars. Very polished. I was impressed. Technically a train wreck with Jehu apparently providing the brains, his description of batteries is comical. Don’t try ANY of that at home.
But I must say that Bream is a talent on a skate board and driving around in the San Diego sun in an all electric Factory Five Racing model 818 and sunglasses is definitely cool – edgy and hip all around.
In my mind, being a 59 year-old multimillionair genius with a gorgeous black wife who has a PHd, a Huey helicopter, a Lear jet and two DC-3’s, ratings to drive them, FIVE gorgeous daughters and a shop to play with electric cars all day, and a bedroom office overlooking a cliff onto the Mississippi river, I’m a couple of light years past cool. I am so cool I can wear yellow shoes and pull it off.
So it’s time to concede pseudo cool to the newbies, and move on. We take it as a sign of the time to move on to a tightened focus at EVTV, though a not unpredictable one. We are going to basically cease promoting electric cars as cool too. I think that is much more established than it was when we started in 2008. We’ll let Starsky and Hutch manage that in the future.
We are going to focus on more modern AC drivetrains, controllers, batteries of course, and the deep electronics needed to integrate that smoothly and effectively in your custom electric vehicle build. We will be, if anything MORE technical and MORE boring to the masses seeking environmental nirvana and perpetual motion.
The only constant is change. And this one is for us, both necessary and foreseen early on having been through it at Boardwatch. Time to tighten the focus and up the game. But I do struggle to picture how the environmentally concerned plug-in america types, along with Nicky Gordon-Broomstick will view our description in this episode of how CAN works between the GEVCU, JLD505 and EVIC – all acronymious enough to glaze anyone over.
This kind of means a sad farewell to some of our loyal viewers but it is just inevitable. I’m just not taking calls anymore on how to get a terminal program up in Windoze. If you don’t know, you should really see EVShow – it’s kind of designed for you. I was sorry to see some of our very ardent BBS operators leave us, and I am truly sorry to lose you too. But we adapt or die. And as the genre grows, we have no aspirations to own it all. We just want the dead center of it where what WE think is the cool stuff happens.
This episode is mostly about what I’ve actually been doing the past two weeks. Battling the GEVCU and CANDue code to set up a simulation of a vehicle with a UQM Powerphase 100 inverter, a JLD505 battery monitoring device, and Brian Gallagher’s brilliant and evolving Electric Vehicle Control Interface EVIC. I can’t wait to get one of these installed in the Yellow VW Thing.
But a battle it has been. To let you in on a little of the back room stuff, I have had the CANdue code crashing horribly for a couple of weeks every time the CAN traffic got up to speed incoming from the GEVCU. And it wasn’t just a matter of finding one of my typos in the code. I determined that very oddly, the system worked perfectly if compiled with Arduino version 1.54 and crashed horribly with version 1.58 or the JUST RELEASED version 1.60.
It took DAYS to discover “this was the problem” and it was a bitter discovery. The 1.60 version cuts a full 20KB of executable from the GEVCU load. Now we are only using 125KB of 525KB available now. But we have to keep an eye on it as the program grows. And to suddenly chop that to 104KB is a big deal. It also takes out a lot of other little bugs and fixes along the way. But it appeared to MURDER my program.
I have to say that Collin Kidder is just becoming key around here. The guy is brilliant. The problem was a lockup when I had debug on and was printing to the screen all the data coming in. The program worked flawlessly without debug. But the whole concept of simulation is that you get to SEE the simulation. He suggested moving to the native USB port from the programming USB port. I thought “fat chance” but tried it anyway. Easy enough to check.
The Arduino Due has two USB ports. One goes through an auxilliary ATMEL chip and uses a serial port on the ARM SAM3X processor. The other is “native” or goes directly in pins on the SAM3X which actually has USB built into the chip. I tried his suggestion with a
#define Serial SerialUSB statement which simply reroutes all printing out the SerialUSB (native) port rather than the Serial (programming) port.
The problem went away.
The question is why? All of our CAN work uses what I think is the most elegant library in all of Arduino-dumb, can_due, which was actually written by Collin. At my urging and behest, he had added interrupts to can_due. He calls them “callbacks” as can_due is interrupt driven anyway. But normally you have to poll for inbound traffic and then if it is available do something with it.
I wanted incoming CAN packets to automatically route themselves by interrupt to the exact function in the program designed to deal with THAT particular CAN message ID. In this way, incoming CAN messages would actually interrupt the main program, process themselves, and then allow the main program to continue.
This allows me to have a kind of a state machine with common variables that to most of my code, appear to simply update themselves in real time by magic. Objects that just exhibit the properly updated state all the time. Communications is actually sealed off behind its own walls. This is a favorite architecture of mine and understand, I’ve been dealing with Serial communications since 1979.
Interrupts, unfortunately, involves its own world of pain and tends to go to the heart of the operating system – in this case Arduino itself. And as Collin states, where interrupts come into play “beyond, there be dragons….” So while he gave them to me in can_due, he doesnt’ actually use them himself.
The problem was, when the interrupt stopped the main program, it allowed the serial buffer to overrun – murdering my program.
I’m not used to having compiler issues. A bit beyond my paygrade. But he somehow determined, initially by guess with me verifying the outcome, that there was a conflict between the programming serial port interrupts and the interrupts can_due was using. As the SAM3X actually supports nested interrupts, this was a puzzle.
He actually had to go into the Arduino code to determine an interesting phenomenon. While the SAM3X supports nested interrupts, Arduino doesn’t use them. Arduino treats ALL interrupts equally. By default, it has them ALL set to the highest priority.
Even more esoterically, he figured out how to CHANGE the interrupt. I don’t think I could figure that out in a hundred years.
So the real fix was to set the CAN interrupt to a LOWER value in can_due and he did it with ONE line of code.
NVIC_SetPriority(m_pCan == CAN0 ? CAN0_IRQn : CAN1_IRQn, 12);
Might as well be magic for all I know. I can’t even read it. A one line fix. I’m totally in awe.
Actually this goes back six months to some strangeness we noticed in the UQM Powerphase object module. It would run for 10 minutes and then just quit. The UQM is a bit odd in that it needs its torque command every 10ms – which is kind of a short time period for CAN messages. Using any compiler after 1.54 we would have this problem so we’ve just been using 1.54 ever since. I have not confirmed this on the bench yet, but I would bet $100 that when I do, I will find 1.60 to work perfectly now – with the new can_due. And of course with a 20kb smaller footprint.
Software is its own world of pain – and joy. I actually love to code but other duties normally call. With GEVCU, I’m kind of driven to do it myself often. And so it occupies more of my time. I would not have THOUGHT that building electric cars would get to be C++ intensive, but that’s where we are circa February 2015. As I’m a little rusty, I would be lost without Collin to go to for adult supervision.
Bottom line, make sure you update your can_due library if you are doing any GEVCU or other CAN code on the Arduino Due.
THAT is about as technical as it will ever get, and not likely to be discussed on the show in any event.
For this episode, I attempted to take the mystery out of CAN and show it in action between three devices with a couple of “CAN busses” (two twisted wires) between them.
CAN is actually very EASY at one level. You have 3-digit message ids. That ID defines what the message IS. Nothing to do with sender or receiver. It defines what it IS.
And then you have eight bytes of data in the payload. That’s what it HAS.
If you have a device that has the knowledge to THROW that message ID, and another device that knows to CATCH it and what to do with the data, you’re done.
And it is important to note that all of that can easily coexist on the same two wires that has LOTS of other devices, and lots of other messages, firing back and forth. Practical limit? Say 30 devices on a 500 kbps bus.
The messages DO have a heirarchical priority in the event of bus contention. The lower the message ID, the higher the priority. So your inverter has 0x200 series messages and your display might have 0x600 series messages. Actually ours do.
If you are doing the software on both ends, it couldn’t be simpler.
Reverse engineering is another matter. There are endless ways to represent things in that eight byte payload. In fact, the entire 64 bits could be bit-encoded. If bit 37 is on, the windshield wipers are on. And if it is 0, they are off. If bit 52 is on, your reverse light is on. If you turn it off, they go out.
We covered two ways to represent negative numbers in two byte integers, as well as how to represent decimal numbers in two byte integers. That’s the sort of thing you run into. When reverse engineering, you have to see the action, and observe the changes in CAN traffic, and then try to devine how it is encoded.
This is kind of like working a crossword puzzle. And it works best in collaboration. In doing the UQM Powerphase, I kind of quickly got to how torque was defined in the command message. Mark Wiesheimer kind of quickly got to a sequence number in the right fourbits of one of the bytes, but it took him some further investigation to determine that the left four bits of the same byte set whether the inverter was enabled at all or disabled, and whether it was in reverse or forward.
With all that worked out, there was a last byte in the payload that varied with each message. And if we didn’t have the RIGHT number in that byte, the UQM would simply not work. We fed commands copied from logs in a captured stream from an operating CODA and it ran perfectly. But if we tried to make up our own torque commands, it wouldn’t work at all.
The reason was we didn’t have the magic security byte at the end. This byte ONLY existed to KEEP us from doing what we were trying to do – run the inverter without being part of the CODA design team. It was a security encryption. A cyclic redundancy check byte that used an external polynomial to calculate the result – a password. Alan Turing being otherwise occupied in the next life, Collin Kidder worked it out.
After he explained how he did it, I still don’t understand how he did it.
And so I see a period of CAN reverse engineering as being the key to using OEM power trains and other devices. And GEVCU and EVIC and JLD505 and other devices as how you actually wire them up and use them.
I’m sorry – an AMR motor at $22,000 is not a solution for someone building an automobile. The concept was absurd when AC Propulsion wanted $25,000 for their drive train. It was absurd when UQM wanted $15,000 for their drive train. It was absurd when we PAID $13,000 for the MES-DEA electric drive train. And it is absurd today. I guess I think it goes beyond absurd actually to insulting. There’s nothing there to account for that price. The devices just don’t cost that much.
If you wreck a Tesla, or a Volt, or a Leaf, or an i3, or a Kia Soul, it is extremely unlikely that the inverter and motor even got the word – much less were damaged by it. And currently, in the used/salvage auto market, these devices are TOTALLY without value. So much so that they are mostly being traded in for the metal value in them. The salvage guys aren’t even pulling them and stocking them for the most part. A few are starting to. But an entire drive train is commonly available for less than $2000 and in many cases you can buy the entire wrecked car for that.
We’re currently charging between $5000 and $10,000 for a 100kw drive train. And it was not TWO YEARS AGO that a 100kw AC drive system for under $10,000 was just UNOBTAINIUM. Roving bands of gypsies wandering the land looking for such a thing – no hope.
And so I would predict a collapse in the price of drive trains, IF we can enable that. It is true we are kind of cutting our own throat here in a way as we derive our operating capital from component sales. But we intend to lead it.
Why? You have to keep your eye on the ball. Our mission is to convert all the cars in the world to electric drive -not to sell parts. We think when we reach the point where all in, you can convert ANY car to electric drive in expert fashion, on par with new electric car operation, for $10,000 in components all in, the market for that will be HUGE. There are millions of beloved models out there begging for restoriation – and new life using electric drive. That’s going to require a bit of a decrease in the cost of batteries as well, but we see a perfect storm over the next three years where that is exactly what is achieved.
This is actually very interesting. If you are going to all the work to re-engine a custom car today, you would never dream of using a salvage engine. First, it’s already dirty. But second, it also has wear on it. AC electric motors just don’t work that way. The difference between a brand new one and one with 10,000 miles on it? You know the one with 10,000 miles on it works. If you are just too fastidious for belief, a set of bearings for the motor runs $50 or so. But I would personally prefer the broke-in bearings myself.
We will of course continue news and developments from OEM land as it sets the tone for the future. This week’s article by the Wall Street Journal noting Apple’s probable entry to electric vehicles caused a strange tsunami in my mailbox. Wasn’t this totally bogus and didn’t I agree? Must be 30 e-mails of that sort. Why would I think it bogus? I predicted it coming on three years ago.
Apple now has $178 billion in the kitty and has not come up with a solution to their biggest problem. In order to be a darling high-tech growth stock, they have to keep growing. And the law of large numbers has them in a Mexican death clutch at the moment.
Here are their gross revenues by year
2014 $182.795 billion – growth 6.95% or $11.885 billion
2013 $170.910 billion – growth 9.20% or $14.4 billion
2012 $156.508 billion – growth 44.58% or $48.249 billion.
2011 $108.249 billion – growth 66.36% or $43.179 billion
2010 $65.070 billion – growth 78.07% or $28.53 billion
2009 $36.54 billion
How can $12 BILLION dollars in increased revenue be a problem? When it represents a growth of 6.95%. This company five years ago was growing at 100% per year. You can see from these numbers the rapid DECELERATION in growth.
True, they added a signficant dividend – basically an admission they DON’T KNOW WHAT TO DO WITH ALL THE MONEY!!!.
To get back to even a MODEST 10% growth they have to find $18.27 BILLION in NEW business this year. Investing in Apple is becoming more of an exercise in buying cash balance than it is high-tech growth investing. And the valuation metrics are VERY different.
To get back to their salad days of 44.58% growth way back in 2012 – they need $76 billion in new business. The ENTIRE automobile business worldwide is only $85 billion if they got ALL of it.
Anything they can acquire in their space that would bring that kind of revenue is an instant visit to Federal court in the U.S and a more aggressive court in Europe on antitrust issues. They badly need an ADJACENT space. That is one that is entirely separate, but will let them use their core competencies to advantage.
And they have $178 billion to spend. Do I believe the story?
I think that if they did as I instructed in 2012, and acquired Tesla, SpaceX and Solar City in a stock swap and immediately pumped $80 billion into it, today they would be on the verge of owning THE satellite network running all of the Internet at gigabit speeds, accessible from anywhere on the planet, and have all three Tesla models in production – mostly at other auto plants in Asia. The Model S would be the original $55K Elon first touted. The Model X about the same. And the coming Gen 3, about now, would be $32,000 – the average car price with a 200 mile range. They would be selling 250,000 cars per year and GM and BMW and VW would be shrieking in terror like eight-year-old girls at a Brownie cookout under attack by boa constrictors.
The Western United States would be entirely solar and moving east at 12 miles per hour.
I don’t think we’d actually be on Mars yet, but if we did eventually go, Apple would own every acre of it.
Steve Jobs basically left his company to an accountant. And that’s just where we are. Seven percent growth and a gold watch – coming in April.
For my money, they are too little, too late, and too lame. But it will STILL be an absolutely industry crushing car.
I’ve never adequately gotten across the chasm of communicating how DISRUPTIVE new technologies are. The concept of the end of land line telephone, spoken in the early nineties, only got me hoots of derision and explanations of how very little and ugly I truly was and how very funny my mother dressed me. That my $5000 360 dot per inch digital camera would do anything to the film camera market was ABSURD in 1994 and I have the very cogent considered e-mail of the experts of the day to prove it.
I say again and only for those with an ear to hear. Picture General Motors and Ford as the walking dead. Toyota – no more – gone in a flash of hydrogen. VW and BMW – goosestepping to irrelevancy. Disruptive new technologies are not very good at live and let live. They don’t coexist with dinosaurs. They cause EXTINCTION. They are toxic to the established order. That’s why we term them “disruptive”.
They couldn’t be touched – way too big to fail:
U.S. Postal Service
Motorola Cell Phone
Full service gas stations with repair bay.
The walking dead now:
All automobile dealers
All book stores
Non-satellite radio stations
The question never WAS whether Tesla Motors and Apple computer can compete in the auto market. It is WHOM among automakers can scramble onto the electric drive raft fast enough to survive. I mean even continue to EXIST under that name. AT&T truly did die. They were broken into seven companies, reconstituted under Bell South, morphed into a cellular company, and nothing of the original company and little of their landline business remains at all.
So you CAN survive as a dinosaur, if you drop the whole scale, teeth, reptile, and roaring noise gig entirely and go in disguise morphing yourself into a bird. But most just die.
For them, and for us, the only constant is change. And the Internet itself has not only accelerated the constant, but is constantly accelerating.