My Kiva effect

 For over a decade, I've been dedicated to contributing to the Kiva movement. Here's the impact I've made so far. As long as I have a job and income, I am committed to continuing this. small help.

2023 annual performace

 Screenshot from account. 

My portfolio heavy in to Bonds.  TLT/TMF.. Monthly OOM call and their premiums.

Hoping for good return this year  ( if FED starts reducing interest rates)

2023 retirement account composition

2023 is a mixed year. ( kind of little early to long term bonds.. both vanguard and TLT... both recovered  a lot.) need to see my luck in 2024. hopefully bond will give much needed returns to my account.

Stocks side, heavy in to energy and healthcare. ( both are duds in 2023.) now all bet in to 2024.

Cash is earning 4.7%.   GAS is looking attractive, will wait till the year end and may make a move.  

This is year 2023. see the website maintenance page

 

Notice: 39% Increase in Monthly Insurance Premiums for Home and Auto Texas

Everything's Bigger in Texas, Including My Insurance Premiums hike:

From $380 to $530 aka 39% Price Increase! 

Waiting on other shocks.

Driving Safer with Smart Driver Scores: An Inside Look at Trip Scoring and Driver Behavior Analysis

 

In my earlier post, I mentioned that I work as a solution architect for a vehicle data platform. In this post, I will discuss how we use trip data information to score driver behavior, using what we call a "smart driver" score. Like a credit score, a driver score is an indicator of a driver's behavior, which can be used for things like usage-based insurance.

The term "smart driver" typically refers to an individual who can optimize their driving behavior to improve safety, fuel efficiency, and comfort. A smart driver might use advanced technologies, such as real-time traffic information or driver-assist features, to make informed decisions about how to operate their vehicle. They might also adjust their driving habits, such as reducing speed, avoiding sudden acceleration and braking, and following a consistent pace, to minimize fuel consumption and emissions.

Smart driver programs and initiatives are often designed to educate drivers about the benefits of safe and efficient driving, and to provide tools and resources to help drivers make informed decisions on the road. Some programs may also offer incentives, such as rewards or discounts, to encourage drivers to adopt safe and efficient driving practices.

In the context of connected and autonomous vehicles, the term "smart driver" may also refer to the vehicle's advanced systems and technologies, which are designed to optimize driving performance and provide a more comfortable and efficient driving experience.

The term "smart driver" is used by a variety of organizations and companies, including automotive manufacturers, government agencies, non-profit organizations, and technology companies.

Automotive manufacturers, such as Tesla and General Motors, may use the term to describe the advanced driver-assist technologies and safety features that they offer in their vehicles.

Government agencies, such as the National Highway Traffic Safety Administration (NHTSA) and the Environmental Protection Agency (EPA), may use the term to describe their initiatives to promote safe and efficient driving practices.

Non-profit organizations, such as the National Safety Council, may use the term to describe their educational programs that aim to promote safe and responsible driving practices.

Technology companies, such as telematics providers and software developers, may use the term to describe their products and services that aim to improve driving performance and provide a more efficient and connected driving experience.

Overall, the term "smart driver" is used to describe individuals and organizations that are focused on optimizing the driving experience, and promoting safe, efficient, and sustainable driving practices.

Core trip concepts, data collection, scoring methodology

When owner is driving vehicle from point A to point B, Ignation ON/OFF cycle,  smart driver system will rate your trip based on various factors such as the duration of the trip, the condition of the road, the traffic, and your driving experience. Here are some examples of factors you might consider when rating a trip:

Drive Time: How long did it take you to complete the trip? Was it faster or slower than expected?

Driving experience: Did you enjoy driving the vehicle? Was it easy to handle, or were there any challenges with maneuvering or controlling the vehicle?

Comfort: Was the trip comfortable? Did you experience any issues with the vehicle, such as discomfort from the seats, or noise from the engine or road?

Road condition: Was the road in good condition? Were there any obstacles, such as potholes or construction, that impacted your trip?

Traffic: Was there a lot of traffic on the road? Did you experience any delays or slowdowns due to congestion?

2023: The Year of Market Uncertainty?

The planet Saturn returns to the same position relative to Earth every 30 years, and it is believed that similar outcomes can be expected as seen in 1993 or 1963. These events may include an abundance of innovation, volatile markets, and instances of death and destruction. This is my first time tracing this phenomenon, and I am eager to see the outcome.

Currently, in USA, we are experiencing various financial conditions, such as high inflation and low employment rates. Need to final outcome.

In 1993, the global financial markets had a positive performance overall. The Dow Jones Industrial Average (DJIA) had a total return of 8.25% for the year, while the S&P 500 had a total return of 4.46%. The NASDAQ also had a positive performance, with a total return of 7.62%. The yield on the 10-year Treasury note also fell during this period, which generally indicates a strong bond market.

In terms of international markets, the Nikkei 225 in Japan had a total return of -0.31%, while the FTSE 100 in the United Kingdom had a total return of 3.57%. The DAX in Germany had a total return of 9.58%.

The U.S. economy continued to expand during this period, with the Gross Domestic Product (GDP) growing by 2.9% in 1993, and the unemployment rate falling to 6.9%. The inflation rate was also relatively low, at 3.0%.

It was worth mentioning that the interest rates were relatively high at that time, with the Federal funds rate at 3.25% by the end of 1993.

Overall, the financial markets in 1993 were characterized by steady growth, low inflation, and a strong bond market.

The Power of Data: Transforming Auto Insurance with Vehicle Management Systems

I am working in vehicle data management system design, development, and support.

While I can not specify fine details, (i.e. data used by specific auto insurance providers), data is being collected heavily and used by many.

1.     Risk assessment: By collecting and analyzing data about a vehicle's usage, insurance companies can better understand the risks associated with insuring that vehicle. For example, data about a vehicle's speed, acceleration, and location can be used to assess the likelihood of an accident occurring.

2.     Premium calculation: Based on the risk assessment, insurance companies can calculate the premium for insuring a vehicle, taking into account the specific risks associated with that vehicle.

3.     Fraud detection: By collecting data from onboard systems and other sources, insurance companies can detect instances of fraud, such as staged accidents or false claims.

4.     Claims management: In the event of a claim, insurance companies can use the data collected by the vehicle data management system to verify the details of the claim and assess the damage to the vehicle.

5.     Usage-based insurance: Insurance companies can offer usage-based insurance plans, which allow drivers to pay premiums based on the actual usage of their vehicles. By collecting and analyzing data about the usage of vehicles, insurance companies can determine the appropriate premium for each driver.

Here are some popular vehicle data signals that are commonly collected and analyzed by vehicle data management systems.

1.     Speed: The speed at which the vehicle is traveling can provide insights into the driving behavior of the driver and the likelihood of an accident occurring.

2.     Acceleration: The rate at which a vehicle accelerates can provide insight into the driving style of the driver and the likelihood of an accident occurring.

3.     Location: By tracking the location of a vehicle, insurance companies can determine the risk associated with insuring that vehicle, based on factors such as the road conditions, traffic density, and likelihood of accidents.

4.     Engine data: Engine data, such as RPM and fuel consumption, can provide insights into the health and performance of a vehicle, which can impact the likelihood of an accident and the cost of repairs.

5.     Brake data: By tracking the usage of a vehicle's brakes, insurance companies can assess the driving behavior of the driver and the likelihood of an accident occurring.

6.     Steering data: Steering data, such as the angle of the steering wheel and the force applied to the steering wheel, can provide insight into the driving style of the driver and the likelihood of an accident occurring.

7.     Fuel data: Fuel data, such as the amount of fuel consumed and the fuel efficiency of a vehicle, can provide insights into the driving behavior of the driver and the likelihood of an accident occurring.

2022 Stock Account Performance

 I used to be very conservative but sometime at the end of april, started expermenting with options & littlebit of crypto. end result is the following.   Need to see the rest of the years. good thing is option income via calls is good i.e. 2 to 3%.  For the rest of the year, i will continue the same. Still some capatial needs to be allocated. Hopefully put it work after Sept quarter/correction.

Sometimes slightly out of box thinking or outlier saves the entire portfolio

 I didn’t like the selection of funds that HAS provider offered me in HAS account.

So, I end up picking pimco commodity fund based on its past dividend returns.

See the result of my portfolio.  All my others account in RED, this is solid green. Amazing run plus great dividends. (May be at the end of Q3, will reduce the position size… I have a feeling, a great downturn ahead of us)

loving excalidraw

 noticed as part of a presentaion & I started using for my upcoming project. 

A case study on how to screw a great product & team

Business and systems architects got good vision on a critical enterprise data product. After some key decisions, everyone worked hard and initial few years later, a good product was deployed/operational.

Some wants to make/take it too great & at the same time, someone in the leadership thought, it is his baby and want to survive on it for rest of his life. so, he hired incompetent leaders to manage people.

This led to constantly delivered contributors are ignored but new commers are promoted. This news alone devastated the key guys. When they realized, they simply moved on within weeks.  Product is still evolving, and this happened. Now a good product became a mediocre product team and quality issues started (this is SAD & yet real story of a company which dreams BIG yet makes bad decisions.)

A  people leader needs to inspire, ( worst case stay neutral) else great turns in to shit quickly.

More later....

Apache Spark JSON pasing confusing errrors

input json:

{

      "shipping_address": {

        "street_address": "1600 Pen Avenue NW",

        "city": "Washington",

        "state": "DC",

         "type": "business",

         "additionalProperties": {

            "test": "one",

             "test1": "two"

          }

      }

}

spark code;

# File location and type

file_location = "/FileStore/tables/mock_example-1.json"

file_type = "json"

# CSV options

infer_schema = "false"

first_row_is_header = "false"

delimiter = ","

# The applied options are for CSV files. For other file types, these will be ignored.

df = spark.read.format(file_type) \

  .option("inferSchema", infer_schema) \

  .option("header", first_row_is_header) \

  .option("sep", delimiter) \

  .load(file_location)

display(df)

Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the referenced columns only include the internal corrupt record column

Solution: usuually spark expects one json message per line..

In general we use Notepad etc, to format JSON examples. ( just to validate the strucrue of the documents)... if U are saving formatted JSON, then spark will fail with the above error.

notepad JSON plugin offers compressJSON option too. so compress/save it. It works fine

Weekday ( Monday) pure EV averages distance driven and electric consumed usage

Based on 50K vehicles, average distance driven/kWh consumed.

In Box  plot form

Kwh consumption (data includes some outliers)

After outliers cleanup

Connected Car Q&A

 Everyday, I work on connected car data projects.

Lately few people repeatedly keep asking what it means. So I put brief Q&A. 

Something I am sharing. (small slice but hoping this is useful)

What is connected Car?

    A car with have access to the Internet and communicate with traditional automotive components, such as the engine and electronics, as well as the smart devices of a driver. All via telematics* system.

What type of car data are talking and how it is useful to a driver or vehicle owner or auto maker or 3ed party?

the most common use of car data is to improve the driving experience by collect the data about driver behavior events i.e., from ignition on to ignition off. This data improves following experiences for the driver.

·        Finding fuel location/battery charging station as needed

·        Local business searches and promotions

·        Journey route weather/traffic updates

·        Real-time data communications about any emergency situations (flat tire etc.), crash etc

·        Location sharing, fast theft response

·        Insurance discounts based on good driving behaviors/usage-based discounts

For vehicle manufactures

a)      Data helps to measure the performance/reliability of the vehicles. Data helps to pinpoint about unforeseen issue(s) with new & old vehicles. Data helps voluntary recall vehicles for specific issues.

b)     Data helps to catch fraudulent warranty clams/odo tampering issues

c)      Various service offering such as oil changes, end of brake pad changes etc.

d)     Offering customer services for example geo fence boundaries for family of drivers. With Teen drivers.

           For 3ed party companies:

     Vehicle location data helps in forecasting about live traffic conditions

    3ed parity insurance providers to offers discounts based on driving behaviors, usage.

      Automatic pothole information improves the road conditions

      Near real time weather data to forecast real time weather

 

Usually Telematic* systems are integrated with Satellite navigation systems and onboard computers and back office systems. Not only data collections, back office controls and refreshes the software inside cars too via over the air updates.

 

Final updates on my Kiva journey

 Following are final statistics. ( ~15 years of journey.)  

Nowadays, this year, I am unable to find anytime.

Following is public profile.

public profile

Nayagara pictures (First time i am able to capture rainbow just before ride)

 

Home Depot Absurd Pricing Example

See the enclosed.
 

A snapshot of EV vehicles location

Generated as part of EdX Python analysis exercise. (generated from of session logs)
 

Hive Timestamp column, default query results behavior, common misunderstanding and formatting to a target zone

    Similar to many RDBMS, Hive Timestamp datatype and it is stored as bigint and are stored as an offset from the UNIX epoch. Usually Timestamps are interpreted to be timezone-less. However end user runs a Hive query (via Hue or any tools), it returns the columns in the default server time zone. (in our case, EST ad team started saying your app is   not saving data in UTC, we need UTC format etc. etc.  After spending sometime with Hive language manual and with some of the past issues, I found an work around via Hive JIRA.

Idea of this query is vehicle sensor data is stored in data table.

Intent of the query is to know what is the value of a sensor on a given day.

select  ID, element_cd,

from_utc_timestamp(to_utc_timestamp(from_unixtime(unix_timestamp(timstm_column) ),'EST'), 'UTC'),

element_val

from  data_elements

where yr_nbr = 2019 and mth_nbr = 11 and day_nbr in (30)

and element_cd='POWER_LEVEL'

Lot’s of indirection. i.e. taking timestamp column & formatting it to unix timestamp and then first converting to default time zone and then to target time zone.  ( Above query returns timestamp value as UTC formatt.  My  HIVE servers are running in EST)