In 2010 I managed to get a work trip to Tokyo. For five weeks I worked with the best people from one NEC department. In this task we were two: Alexey M. and me.
Alexey, thank you for your support and connection on the trip. It was prudent of you to bring a spare 3G phone with a tourist SIM card. ありがとございました。(arigatō gozaimasu)!
My task was to assemble two server configurations, modified by NEC, with an Oracle database installed:
- 4 blades running in parallel in RAC (Real Application Cluster) configuration
- 4 blades connected by a special bus as one single entity (Single Oracle Instance)
The tests were optimized for parallel queries and we expected to see impressive results on the most modern equipment in the first configuration.
In Fuchū, Tokyo (東京都), our Japanese department head—an older gentleman with gray hair himself—personally connected cables inside the servers, not allowing anyone from my team to get close. Both sides were anxious about this work, so top forces were deployed and everything was done in the best possible way.
[
][Wikipedia link]
We started with the second configuration. During testing we obtained respectable results (I don’t remember the exact numbers). We repeated it several times and then switched the servers to RAC cluster mode.
Days went by, but the tests kept showing a multiple disadvantage for RAC versus the single instance. Moscow colleagues repeatedly rewrote the test scripts, yet we could not get close to matching the performance of the second scheme. The results were never shared with us.
Our Japanese colleagues noticed our embarrassment. On day 32 of the trip, after taking responsibility for the tests myself, I decided to approach their management and admit that according to our measurements, the RAC configuration was significantly inferior to the Single one. No amount of kernel or filesystem optimizations (RHEL/ext3) could improve this.
The elderly Japanese gentleman listened politely to my report right next to the server. He then called everyone else in and asked them all to repeat what he’d just heard. Everyone laughed together with him. And he said:
“These results are exactly what we expected. The corporation had modified standard servers so that their processors and memory would work together. We were waiting for your tests under heavy load; otherwise, we didn’t have the opportunity ourselves to test this. That’s why we’re very happy with the results—all our expectations from the new servers were met. Thank you!
What relief came over me at that moment! The large company I represented simply didn’t care about what was going on inside me and, in fact, didn’t give a damn how I chose to represent it by coming forward with my admission of defeat. But apparently this is how things go. That a bad result turned out to be an expected result.
The director and his assistant friend invited Alexey and me for lunch at a small restaurant opposite the data center. I think offering us to try food from their plates was a sign of the highest respect and hospitality.
Japanese people don’t abuse their high positions. Although the gate on the exit area was open, we waited three minutes until 14:00 so that we could exit together with everyone else.
This is the most impressive trip I’ve ever had. Interacting with people from Japan at that time opened a lot of new things in me. Initially I expected UFOs at Narita airport, then looked for them on the streets. But instead I encountered a Toyota Crown taxi and angular cars—ordinary tractors and trucks. The train Narita Express got stuck in the rain and only managed about 40 km/h, while the subway runs with clockwork precision, and each line has trains that produce their own unique sound. Japanese people are obsessed with railways. There is far more than just “JR” in JR compared to RZD.>In my work plan I learned about an approach that led the industry to stability. Now it may be that something has changed, and they say Toyota isn’t what it used to be. But it was from there that the expression originated: “first you have to train good employees, then you can build good cars” — and Japanese companies started the Lean movement—lean manufacturing. Many of its traits are applicable to software development. Here we have the same assembly lines and people. I want to ground my work in a Lean and Gemba approach, which I’ll soon start covering in a series of articles and share practical experience about how it works.
And finally:
