A case study on how to screw a great product & team

Business and systems architects got good vision on a critical enterprise data product. After some key decisions, everyone worked hard and initial few years later, a good product was deployed/operational.

Some wants to make/take it too great & at the same time, someone in the leadership thought, it is his baby and want to survive on it for rest of his life. so, he hired incompetent leaders to manage people.

This led to constantly delivered contributors are ignored but new commers are promoted. This news alone devastated the key guys. When they realized, they simply moved on within weeks.  Product is still evolving, and this happened. Now a good product became a mediocre product team and quality issues started (this is SAD & yet real story of a company which dreams BIG yet makes bad decisions.)

A  people leader needs to inspire, ( worst case stay neutral) else great turns in to shit quickly.

More later....

Apache Spark JSON pasing confusing errrors

input json:

{

      "shipping_address": {

        "street_address": "1600 Pen Avenue NW",

        "city": "Washington",

        "state": "DC",

         "type": "business",

         "additionalProperties": {

            "test": "one",

             "test1": "two"

          }

      }

}

spark code;

# File location and type

file_location = "/FileStore/tables/mock_example-1.json"

file_type = "json"

# CSV options

infer_schema = "false"

first_row_is_header = "false"

delimiter = ","

# The applied options are for CSV files. For other file types, these will be ignored.

df = spark.read.format(file_type) \

  .option("inferSchema", infer_schema) \

  .option("header", first_row_is_header) \

  .option("sep", delimiter) \

  .load(file_location)

display(df)

Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the referenced columns only include the internal corrupt record column

Solution: usuually spark expects one json message per line..

In general we use Notepad etc, to format JSON examples. ( just to validate the strucrue of the documents)... if U are saving formatted JSON, then spark will fail with the above error.

notepad JSON plugin offers compressJSON option too. so compress/save it. It works fine

Weekday ( Monday) pure EV averages distance driven and electric consumed usage

Based on 50K vehicles, average distance driven/kWh consumed.

In Box  plot form

Kwh consumption (data includes some outliers)

After outliers cleanup

Connected Car Q&A

 Everyday, I work on connected car data projects.

Lately few people repeatedly keep asking what it means. So I put brief Q&A. 

Something I am sharing. (small slice but hoping this is useful)

What is connected Car?

    A car with have access to the Internet and communicate with traditional automotive components, such as the engine and electronics, as well as the smart devices of a driver. All via telematics* system.

What type of car data are talking and how it is useful to a driver or vehicle owner or auto maker or 3ed party?

the most common use of car data is to improve the driving experience by collect the data about driver behavior events i.e., from ignition on to ignition off. This data improves following experiences for the driver.

·        Finding fuel location/battery charging station as needed

·        Local business searches and promotions

·        Journey route weather/traffic updates

·        Real-time data communications about any emergency situations (flat tire etc.), crash etc

·        Location sharing, fast theft response

·        Insurance discounts based on good driving behaviors/usage-based discounts

For vehicle manufactures

a)      Data helps to measure the performance/reliability of the vehicles. Data helps to pinpoint about unforeseen issue(s) with new & old vehicles. Data helps voluntary recall vehicles for specific issues.

b)     Data helps to catch fraudulent warranty clams/odo tampering issues

c)      Various service offering such as oil changes, end of brake pad changes etc.

d)     Offering customer services for example geo fence boundaries for family of drivers. With Teen drivers.

           For 3ed party companies:

     Vehicle location data helps in forecasting about live traffic conditions

    3ed parity insurance providers to offers discounts based on driving behaviors, usage.

      Automatic pothole information improves the road conditions

      Near real time weather data to forecast real time weather

 

Usually Telematic* systems are integrated with Satellite navigation systems and onboard computers and back office systems. Not only data collections, back office controls and refreshes the software inside cars too via over the air updates.

 

Final updates on my Kiva journey

 Following are final statistics. ( ~15 years of journey.)  

Nowadays, this year, I am unable to find anytime.

Following is public profile.

public profile

Nayagara pictures (First time i am able to capture rainbow just before ride)

 

Home Depot Absurd Pricing Example

See the enclosed.
 

A snapshot of EV vehicles location

Generated as part of EdX Python analysis exercise. (generated from of session logs)
 

Hive Timestamp column, default query results behavior, common misunderstanding and formatting to a target zone

    Similar to many RDBMS, Hive Timestamp datatype and it is stored as bigint and are stored as an offset from the UNIX epoch. Usually Timestamps are interpreted to be timezone-less. However end user runs a Hive query (via Hue or any tools), it returns the columns in the default server time zone. (in our case, EST ad team started saying your app is   not saving data in UTC, we need UTC format etc. etc.  After spending sometime with Hive language manual and with some of the past issues, I found an work around via Hive JIRA.

Idea of this query is vehicle sensor data is stored in data table.

Intent of the query is to know what is the value of a sensor on a given day.

select  ID, element_cd,

from_utc_timestamp(to_utc_timestamp(from_unixtime(unix_timestamp(timstm_column) ),'EST'), 'UTC'),

element_val

from  data_elements

where yr_nbr = 2019 and mth_nbr = 11 and day_nbr in (30)

and element_cd='POWER_LEVEL'

Lot’s of indirection. i.e. taking timestamp column & formatting it to unix timestamp and then first converting to default time zone and then to target time zone.  ( Above query returns timestamp value as UTC formatt.  My  HIVE servers are running in EST)

A diversified portfolio: 5% Inflation-Adjusted Income & moderate growth For Life

Usual concept is to hold 60% in stocks and 40% in bonds. While there are different methods/selections in to which stocks/bonds, I am using following allocations in 401K for each ongoing monthly contributions. Following are list of funds with contribution amounts.

     VEMAX( Vanguard Emerging Markets Stock Index Fund Admiral Sh )  5%

   VGENX ( Vanguard Energy Fund Investor Shares )   5%

   VGHCX ( Vanguard Health Care Fund Investor Shares ) 20%

   VGSLX ( Vanguard Real Estate Index Fund Admiral Shares) 10%

   VFIAX ( Vanguard 500 Index Fund Admiral Shares) 20%

   VWINX    Vanguard Wellesley Income Fund Investor Shares  (20%)

   VUSTX   Vanguard Long-Term Treasury Fund Investor Shares (20%)

No	Fund name	Annual dividends
1	VEMAX	4
2	VGENX	4
3	VGHCX	10
4	VGSLX	6
5	VFIAX	6
6	VUSTX	4

                                         Total 34/6 ~6%

Again I am not reinvesting all dividends automatically. All dividends sits in Vanguard money market account  with 1.X% monthly return. Again my intent is use this money in case of a deep corrections in the markets.  So far, last year above portfolio gave 5K dividends and moderate growth of 8%. Again last year is an exception with all time highs, but for past decade, my other portfolio is returning above average returns.  In my other portfolio, I am using 80% stocks and 20% bonds. ( other one is IRA and I am using little bit individual stocks too.  But same set of funds with different compositions.

The possible range of expected annual portfolio returns for the given period is 7.11% to 12.15%. 

I am not a financial analyst. I am not an expert in money, monetary policy, investment theories, theorems. My blog article should not be taken as advice, financial or otherwise, or as a statement of the right or proper way to do things. What I am is a regular IT guy, middle-aged, with a life full of past mistakes, most of which, I hope, I have learned enough from to continue forward in a positive direction. 

A long term IRA Portfolio Journal

I made plenty of mistakes my first year, primarily through trading and not investing. Again I joined stock markets race during Y2k melt-up situations. Everyone is talking about stocks during break time & this is first job in USA & took trading route. Just to make some quick bucks here.  But after 15K losses, With marriage, my philosophy has changed. All investing is limited to spouse IRA and my 401K only.  It essentially remains thus: build a core of 5 diversified funds with low expense ratio and 10 dividend paying equities, reinvest the dividends and allow them to grow. I don't have a lot of hard and fast rules for my holdings other than they must pay at least 5%. Also if a stock/fund give more than 25% of profits, sell 50% of position and lock the profits.

Currently, the retirement Portfolio consists of five funds and 20 odd stocks. Most of the stocks picks are 5 years old. I am holding some of these funds for 15+ years.

List of funds are:

Ø  VEMAX( Vanguard Emerging Markets Stock Index Fund Admiral Sh )

Ø   VGENX ( Vanguard Energy Fund Investor Shares )

Ø   VGPMX ( Vanguard Global Capital Cycles Fund )

Ø   VGHCX ( Vanguard Health Care Fund Investor Shares )

Ø   VGSLX ( Vanguard Real Estate Index Fund Admiral Shares)

Ø  VFIAX ( Vanguard 500 Index Fund Admiral Shares)

List of stocks includes

BEP

BIP

AMZA

REML

DX

CHI

GUT

ARCC

CII

EXG

APLE

BGR

RRC

PCI

AM

VEON

BGCP

AWP

CHMI

AMLP

GDX

I am not a financial analyst. I am not an expert in money, monetary policy, investment theories, theorems. My blog article should not be taken as advice, financial or otherwise, or as a statement of the right or proper way to do things. What I am is a regular guy, middle-aged, with a life full of past mistakes, most of which, I hope, I have learned enough from to continue forward in a positive direction. I am certainly not an expert in anything, but I am willing to learn, to ask right questions and take some risks.

At the end of December, the IRA is 20 years & I portfolio returned ~4.5% over last 20 years. But my point with this blog spot is long term view really works.  Regularly invest from paycheck and diversify and stay calm and ignore all the noise. Also always keep 5% of money in cash form. Sometime, if you find real value in anything, this helps.

2019 Summer Everglades summer trip and some advice

Will post entire trip details this week.

But for now, i will start with few pictures.

What's the biggest financial mistake you see many people make all the time?

Back when I was 28 years old, year 2000, I was making nearly ~100K minimum a year in IT field.

Every weekend, visiting A place for entertainment & spending, buying cool gadgets(At some point 3 TVs, Desktop, laptops, half dozen audio players etc.) On tops, based on  TV news, jumped in to Y2K stock market madness. But when stock market crashed, newly married, after a loss of job with 10K credit card debt, and on H1B Visa. Many sleepless night for short few weeks.

Luckily I found a job & saved me the entire scene.

Thinking backwards, I acted so dumb. Buying stocks based on news ( with margin). In my case, i used credit card money. plane DUMB.  So much of stuff for no reason.

fast forward 2019……..

Today, I am more comfortable.

 I learned how to save, invest, buying absolutely required stuff. More later

Funny amazon prime deals ( so far)

See the pictures below.
Why in the world, someone buys 3LB?
Jackup price and give 35% off.

My first Rome trip (between July 3ed to 6th) Day-1 & 2

Following is our complete trip summary

1) 3ed. We started from London Gatwick Airport and reached Rome around 10.30

I made earlier hotel reservation around  Vatican . So i at the airport,  we are walking towards bus station. However, on the way,  i nice shuttle operator stopped me and explains, i will offer $10 Euro ride to Vatican which cheaper than the train option. So i took Shuttle service for $40 ( family of 4)
We reached our apartment place. After quick refresh/baths and quick lunch ( yummy Pizza) we headed to Vatican  Museums
( i made reservation when i was USA, Following is the website. https://biglietteriamusei.vatican.va/musei/tickets/do_

Since, i have reservation, i simply walked directly to the entrance. 
( if you dont plan a head, then you will struck in long lines for 2+ hours. Lines under direct sun light.
Outside, i noticed lot's of people advertising, no lines. direct check etc for extra $)

Lot's of rush but we managed to enter the Museum by 2.30 PM. ( my actual timing was 3PM. But no one care. So many people, all they care is the entrance ticket)

Few advice to family folks. ( before entrance, eat a lot. + carry lot's of liquids. Water etc. 
During our visit time, temperature is around 33 degree C)

We stayed 3 hours in the  ( there is a short cut to reach  Vatican Basilica by skipping 40% of museum. But we choose to watch all rooms as much as possible.)

around 7PM we reached our hotel room. this 10 minutes walk to Vatican/Vatican Basilica)
After some refresh, we went to dinner.  Again pizza/ drinks for dinner.)

We wrapped our first day like this

2ed Day:
by 7.30 we finished our breakfast and walked to St. Peter's Basilica)
No many people even on Wednesday 7.30 AM.
We did spend quality time in the St. Peter's Basilica and outside. Nearly two hour gone just like that.

Later we walked to nearby but-stop and tourist center to purchase Roma passes. ( these centers are open from 9.30 AM.   Caution: they will not accept Cash So you have use Credit card only. For parent we bought Roma passes and for kids, i took 3 day bus passes only.  Kids under 18 are free for most of the attractions in our list. So we choose Roma passes for parents only.)
After that we picked a bus to Termi station and later one bus to catch Triva fountain.  (Initially finding bus numbers/stop is little tricky. But my older son cracked the code. Will write more on that topic later.)

2018 summer vacation UK/ROme

Location:British Museum

Interesting  idols from INDIA/south Asia  gallery

.

2018 Summer UK/Rome trip cntd 2

Location:Stonehenge   Day-3

Will add more later