JP Morgan Chase's Hadoop system. Has the time arrived to review Big Data analytics infrastructure?

 

JP Morgan Chase's Hadoop system. Has the time arrived to review Big Data analytics infrastructure?

Dr. Prasad Kulkarni- Consultant- Britts Imperial University Sharjha.

 

Hadoop

 

It's an open source platform used to solve big data using a network of computers. It provides a framework for distributed storage and processing of big data using the Mapreduce programming model.  It is built on the principles of framework should handle the common hardware problems.  Hadoop is versatile and can be used for different tasks. More than this Hadoop offers scalability to enterprises.

Hadoop in the financial Sector

Risk Modeling

Financial sector faces the problem of credit risk, market portfolio risk, and operational risks.credit risks are subject to the bankruptcy of debtors whereas market portfolio risks arise due to inverse returns in portfolio returns. Further, operational risks emerge due to failure of organization’s processes[1]. These risks made financial sector enterprises maintain multiple databases. However, for analysis assembly of files into a  single repository is necessary. This resulted in financial sector companies trusting on Hadoop.

Mass customization:

Financial sector is becoming competitive. To be in the forefront, financial services firms are personalizing their offers. A few firms took a step ahead to offer customized products to clients. This mass customization effort coupled with risk analysis required a quality repository that Hadoop can support.

 

Local data warehouse v/s cloud data warehouse

The year 2008, is the turning point in the financial sector. Till then banks were over protecting their customer data in the local data warehouse. It resulted in the financial crisis as many had no idea about customers other than data submitted to banks. These financial institutions began unearthing the information from emails sent by costumes, their call center conversations, and chat sessions with company representatives. The data produced was enormous for financial services firms. A few of them adopted Hadoop Mapreduce for sentiment analysis, text analysis and behavioral analysis using cloud solutions.

Market predictions

A financial firm has to keep a tab on the stock exchanges how the company and its competitors are performing. It also keeps vigil on regulatory bodies and their change in policies that may affect the firm in the future. The challenge was these data sources were independent and needed integration. Hadoop has worked on integration of these sources to provide more clear insight about the market for financial services firms.

About J.P.Morgan chase

J.P morgan Chase is the largest bank in the USA, headquartered at NewYork city. The bank is named in the Fortune 500 list at the 24th position. It provides investment banking and financial services[2] The former chemical bank merged with Chase Manhattan  corporation in 2000 and was renamed as the J.P. Morgan Chase. JPMorgan's business consists of four main segments: Consumer and Community Banking, Corporate and Investment Banking, Commercial Banking and Asset Management. J.P Morgan Chase built on principles of providing exceptional customer service with integrity and responsibility. J.P.Morgan Chase is a leader in investment banking, financial services for consumers and small business, commercial banking, financial transactions processing and asset management[3].

J.P.Morgan Chase Hadoop system

Credit card information:

J.P.Morgan Chase collects huge data from credit cards of customers. The unstructured data is supported by the Apache Hadoop framework.

Customer service information

The company gets a large number of emails from customers . Though it puts in a relational database but started using an open source framework. It helps the company to do proper risk management[4]

Challenges to Hadoop in 2022.

Emergences of new technologies

Hadoop which was a replacement for relational databases is facing stiff competition from Spark which is internal memory based.  Further, AWS and Microsoft Azure provide cloud based service with faulty tolerant distributed computation at affordable price.

Smile file problem

Hadoop was developed for large files. As we are entering the world of specialized niche software, files may be smaller but Hadoop Mapreduce can not handle data less than 128 MB[5].

Real Time analytics

Hadoop works on batch processing. Hence it is very slow in processing. Developers are mounting Spark on Hadoop systems to get real time analysis[6].

Path ahead for J.P Morgan Chase.

J.P. Morgan Chase  began using Sqrrl, a market app collecte tax saving data, SIP mutual funds and goal based investments for big data analytics. It integrates different datasets and offers data security. The app works on graph analytics to find any outliers in the data security[7].

J.P morgan availed the services of Palantir . This big data analytics software integrates unstructured and structured data to improve the search capabilities. Further, the software helps J.P. morgan chase to to integrate the data and qualitative analytics. This helped the company to identify the  internal fraud in the company[8].

Datawatch another application in big data analytics leveraged by J.P.Morgan Chase to predict market information. It developed on the Data watch platform to keep tabs on real time data arising on the web pertaining to domains of J.P Morgan chase. And offer solutions to the company.

The adoption of niche applications  and emerging trends in Big data technology raised following questions to J.P. Morgan

1.      To continue or not with the Hadoop system for the future?

2.      Is there a necessity for mourning Spark on Hadoop for faster and real time analytics?

3.      Does the company continue to use niche software for specialized functions and create data platforms or look out for third party vendors offering end to end solutions?xctionaclient service; acting with integrity and responsibility; and supporting the growth of our employees.

eliv

References:

1.      Risk Modeling. (2022). Retrieved 11 October 2022, from https://nms.kcl.ac.uk/reimer.kuehn/riskmodeling.html

2.      JPMorgan Chase - Wikipedia. (2022). Retrieved 11 October 2022, from https://en.wikipedia.org/wiki/JPMorgan_Chase

3.      Our Business. (2022). Retrieved 11 October 2022, from https://www.jpmorganchase.com/about/our-business#:~:text=We%20are%20a%20leader%20in,transactions%20processing%20and%20asset%20management.

4.      How JPMorgan uses Hadoop to leverage Big Data Analytics?. (2022). Retrieved 11 October 2022, from https://www.projectpro.io/article/how-jpmorgan-uses-hadoop-to-leverage-big-data-analytics/142

5.      The Good and the Bad of Hadoop Big Data Framework. (2022). Retrieved 11 October 2022, from https://www.altexsoft.com/blog/hadoop-pros-cons/

6.      Chaturvedi, V. (2014). When to and when not to use Hadoop. Retrieved 11 October 2022, from https://www.edureka.co/blog/5-reasons-when-to-use-and-not-to-use-hadoop/

7.      Bajaj, K. (2018). Sqrrl: This free app will solve all investment related confusion. Retrieved 12 October 2022, from https://economictimes.indiatimes.com/magazines/panache/sqrrl-this-free-app-will-solve-all-investment-related-confusion/articleshow/62862969.cms?from=mdr

8.      Top 6 Big Data and Business Analytics Companies to Work For in 2022. (2022). Retrieved 12 October 2022, from https://www.projectpro.io/article/top-6-big-data-and-business-analytics-companies-to-work-for-in-2015/107

 

 

A case study on Tesla shared autonomous cars: Challenges of Artificial intelligence in Robotaxis.

 A case study on 

Tesla shared autonomous cars: Challenges of Artificial intelligence in Robotaxis.

Dr. Prasad Kulkarni, Consultant, Britts Imperial University, Sharjah. 01/10/2022

• Tesla history

• Artificial Intelligence in Tesla cars

• Autopilot, Hydranet, Pytorch, Dual Ai chips, crowdsourcing, imitation learning and tracking systems.

• Overview of robotaxis and Tesla’s  robotaxis

• Current challenges of Tesla Robotaxi

On 30 September 2022, Mr. Elon Musk, a legendary CEO of Tesla, was in his Austin, Texas office meeting with senior executives of the company. He  was excited about the big announcement for Tesla AI day. These include humanoid , Teslabot, self-driving cars and DoJo Ai chips. However, the criticism from the public for not adhering to the announcement dates was in his mind. The competition arena is also heating up as rivals Waymo and Cruise got licenses to operate robotaxis in a few states of the USA. There are glitches found in computer vision technology used by Tesla. Thus it put the company in ambiguity to use fully computer vision or couple with LIDAR. Hardware sharing and understanding the driver behavior raised the privacy issues. However, the company is committed to offer Fully Self Driving(FSD) cars commercially in the selected areas of the USA.

About Tesla:

Tesla is the market leader in electric vehicles and clean energy products. Its market capitalization has reached a staggering $840 billion in 2021. As a result, Tesla has become one of the most valuable global automobile conglomerates. It acquired 21% of the global battery electric market and 14% of the plug-in market. Tesla has supplied a massive number of battery energy storage systems, touching 4 gigawatt hours(GWh) in 2021.  The saga of Tesla's development is interesting. It was incorporated in July 2003 by Martin Eberhard and Marc Tarpenning . The company is named after the inventor, Nikola Tesla. However, the company re-engineered its image after Mr. Elon Musk took over as CEO in 2008. The major objective of Tesla is to provide sustainable transport and energy. 

Artificial intelligence in Tesla

Figure 1: Tesla Model 3 sensors and computing.

Autopilot

Elon Musk first announced the Autopilot project in 2013. A year later, in 2014, Tesla offered customers the opportunity to pre purchase the Autopilot in association with Mobileye. 

Figure 2: AI autopilot Demo

The Autopilot project was developed on deep neural networks. It consists of sensors, radars, and cameras. For any automobile manufacturer, driver safety is the foremost requirement , and Tesla is no different. It uses ultrasonic sensors to detect moving and stationary objects. Furthermore, it identifies the proximity of the object.  

Tesla cars widely used computer vision technology. Tesla cars have rearward-looking side cameras, trunk handle cameras, forward-looking cameras, and triple front cameras. These captured videos were passed through machine learning algorithms. Further, the data uses convolution neural networks for object tracking and detection.

Radars are important for detecting nearby vehicles and objects to avoid possible collisions. These radars are tested in different weather conditions to ensure error free services. 

Hydranet:

Tesla is a pioneer in using neural networks. However, neural networks become expensive when a vehicle is stationary. Hence, Tesla ran the computer vision processes on the ResNet-50 shared backbone. This neural network shared backbone is known as Hydranet. The information processed on the hydranet is recurrent. The traffic signal images, pedestrian images, or lane changing images are recurrent . These instances required a few parts of the neural network. 

Figure 3: Hydranet architecture

(Source: fireblaze aischool)

Hydranets perform the following functions: road markings, traffic signal management, pedestrian crossings, number of pedestrians, overhead signs, neighboring vehicles, static objects, and environmental tags. Tesla has 8 sensors/cameras to support hydranet. There are eight hydranets performing the eight different tasks mentioned above. 

Pytorch:

Facebook's AI research lab (FAIR) popularized Pytorch.  Tesla's computer vision neural networks were trained on Pytorch. Unlike competitors, Tesla doesn't use LIDAR and purely relies on computer vision. The Pytorch tasks include: workflow scheduling, calibration of model threshold, simulations, and passive tasks. 

Dual AI chips:

The electric vehicle industry is in a nascent stage. In such industries, experimentation is a common task. However, commuters' safety is also paramount to Tesla. This has exerted pressure on Tesla to use two AI chips. In the event of one chip failure, the other chip continues to work. This ensures a smooth ride for Tesla cars for commuters. To support the mesmerizing journey, the AI chip of Tesla comprises 6 billion transistors. These chips with 32 MB of static RAM(SRAM) memory are faster and cheaper than competitors. Hence, collecting data by Tesla is faster compared to Dynamic RAM(DRAM). 

Crowdsourcing and imitation learning:

Tesla vehicles run across the world. These vehicle sensors send an enormous amount of data to the company. So Tesla could understand the driver's behavior and situations in which the data is generated. This artificial intelligence-based study helped Tesla algorithms learn the machine and driver behavior patterns. This type of learning is popularly termed as "imitative learning'' in Tesla. 

Tracking system:

Tesla is way ahead of competitors in technology implementation. The company stores the incorrect data arising from vehicles to train the neural networks. Thus, ensuring future models do not exhibit this behavior. It also tracks driver behavior in the transit. If the driver is idle for a long time, a message is delivered to alert the driver. 

Robotaxi:

These are driverless taxis operated by ridesharing companies. These cars developed on electric vehicle technology opening a new branch of study called transportation as a service(TaaS). Baidu announced the radio taxii cars will be available for $77000. This brings the hope of scalability. Another notable company Waymo expects the hardware cost to go at $ 0.30 per mile. Currently, robotaxi service providers are testing the car in geo fencing areas. These areas are labeled as Objective Design domain(ODD) in the robotaxi industry. Waymo and cruise got a license to run their radio taxi in the California state of the USA. Similarly, Baidu and pony.ai got licenses to run radio taxis in China. The maiden trial was tested in April 2016 by MIT in collaboration with Nutonomy. They worked on Renault Zoes to get initial responses. The encouraging results made Grab a southeast Asia car sharing company to have tie up with Nutonomy. 2017 turned out to be the major year for robotaxi industry. In March 2017 Uber tested robotaxis in Pittsburg and waymo began testing its taxis in phoenix Arizona. The same year cruise announced radio taxi service for its employees. In February 2021, Waymo invited the public to apply to test the radio taxi service in the limited areas with its engineers assisting the car. In february 2022, cruise opened up Robotaxi service in california for the public. 

Tesla’s  Robotaxi

The Tesla learning curve in electric vehicles and autonomous vehicles has helped it to implement Robotaxi. Unlike its Chinese and American counterparts, Tesla allowed drivers to learn to drive the vehicle from the beginning. The decision was an outcome of different road conditions and safety requirements. 

In April 2022, Elon Musk announced that the company was building a futuristic vehicle for the robo taxi industry. The car will be built on full Self Driving( FSD) and won't have a steering wheel or pedal. Further, the company work on the principle of cost per mile should be optimum

Figure 4: Proposed model of Tesla Robotaxi

The Tesla robo taxi will be unveiled in 2023 and its mass production will begin in 2024. The car has an office like structure wherein a customer can start working as soon as he gets into the car. The taxi also has ample space for customers to sleep. Elon musk in his recent interview during the opening of the Austin factory in April 2022 said” “With respect to full self-driving, of any technology development I’ve been involved in, I’ve never really seen more false dawns or where it seems like we’re going to break through, but we don’t, as I’ve seen in full self-driving. Ultimately, what it comes down to is that to sell full self-driving, you actually have to solve real-world artificial intelligence, which nobody has solved. The whole road system is made for biological neural nets and eyes. And so actually, when you think about it, in order to solve driving, we have to solve neural nets and cameras to a degree of capability that is on par with, or really exceeds humans. And I think we will achieve that this year.” It was evident from the speech that computer vision and neural networks have a significant role to play. 

Tesla Robotaxi may be used for rental services in the future. The car can be used as mobile suites in travel and there was evidence from China wherein this concept was implemented. More than this, charging Tesla cars is cheaper than using gasoline based cars. In an interesting description Elon Musk pointed out that Tesla car owners use their cars for 12 hours in the week. Another 20-25 hours in a week customers can rent out Tesla cars for Robotaxi and earn extra revenue.

Tesla Cars currently working on Level 2 certifications as they require driver assistance. The company has to achieve Level 5 to get the license from the USA authorities to run robo taxis. 

Challenges

Energy consumption

The energy consumption of the redistribution of empty vehicles is a critical challenge for autonomous taxis. Waymo, an early entrant in the robo taxi industry, had only 8% occupancy in California. For the remaining time , the taxi was loitering and consuming more energy. 

Unresolved and dangerous technical problems

Robotaxis are expected to reduce traffic problems and bring down the cost of commuting. However, the Cruise in San Francisco caused traffic problems by not detecting the congestion and traffic lights properly. 

Hardware and design issues

Elon Musk's plan of Radio Taxi suffers from two serious limitations. First, it uses very old hardware in their existing system, and second, it doesn't have space for LIDAR. The roof in Tesla's robotic taxi is made transparent. If Tesla wishes to use LIDAR, it should change its hardware and software. Tesla can not launch robotaxis from existing cars as computer vision technology alone is not enough to run fully self-driving cars.

AI mapping

Tesla may train its Dojo chips to capture the data using deep learning.  However, the system may make mistakes when collecting useless data. For instance, a driving car might collect animals nearby that are not required by the autonomous car systems..

Neural network modeling

The Tesla artificial intelligence system has become a hard problem for the company. The company generates a lot of data but is unable to identify the training data a few times. Another issue that popped up in Tesla was the need to define the neural network parameters to test the efficiency of the system. Though Tesla worked on multitasking feature sharing and task decoupling, the problem is still unsolved. 

Conclusion:

Robotaxi will prosper all over the world. The accenture survey of robotaxis had 49% customer acceptance. However, there are two contradictory concepts evolving namely computer vision based and Hardware LIDAR based. Tesla working on fully computer vision based technology is lacking behind in achieving the FSD readiness like Waymo and Cruise. AI is evolving and needs time for Robo taxi companies to provide complete FSD services. On the flipside, human privacy and car hardware compromise may raise serious issues in the future. 

References.

1. How Tesla uses AI and CV - Blogs | Fireblaze AI School. (2021). Retrieved 30 September 2022, from https://www.fireblazeaischool.in/blogs/how-tesla-uses-ai-and-cv/#:~:text=3.-,AI%20Chip%20%E2%80%93%20Dual%20Chip%20System,work%20through%20the%20spare%20units.

2. Marr, B. (2021). How Tesla Is Using Artificial Intelligence to Create The Autonomous Cars Of The Future | Bernard Marr. Retrieved 30 September 2022, from https://bernardmarr.com/how-tesla-is-using-artificial-intelligence-to-create-the-autonomous-cars-of-the-future/

3. Lanctot, R. (2022). What’s Wrong with Robotaxis? - Semiwiki. Retrieved 30 September 2022, from https://semiwiki.com/automotive/316123-whats-wrong-with-robotaxis/

4. Alvarez, S. (2022). Tesla formally lists Robotaxi as part of vehicles "in development." Retrieved 30 September 2022, from https://www.teslarati.com/tesla-robotaxi-in-development/#:~:text=More%20comments%20about%20the%20upcoming,steering%20wheels%20or%20pedals%20anymore.

5. Aguirre, J. (2022). Everything we know about the Tesla Robotaxi. Retrieved 30 September 2022, from https://www.notateslaapp.com/news/755/everything-we-know-about-the-tesla-robotaxi

6. Templeton, B. (2022). Tesla Teases A Custom Robotaxi, Are They Crazy?. Retrieved 30 September 2022, from https://www.forbes.com/sites/bradtempleton/2022/04/21/tesla-teases-a-custom-robotaxi-are-they-crazy/?sh=53af9aa65eb5

7. Greene, T. (2021). Neuralink and Tesla have an AI problem that Elon’s money can’t solve. Retrieved 30 September 2022, from https://thenextweb.com/news/neuralink-tesla-have-an-ai-problem-elons-money-cant-solve

8. Robotaxi - Wikipedia. (2021). Retrieved 1 October 2022, from https://en.wikipedia.org/wiki/Robotaxi

9. Merano, M. (2022). Tesla Robotaxi to shake up rideshare and hospitality industry: Opinion. Retrieved 1 October 2022, from https://www.teslarati.com/tesla-robotaxi-uber-airbnb/

10. 2024, E. (2022). Elon Musk Makes Fresh Claim about Tesla Robotaxi, Saying Production to Start by 2024. Retrieved 1 October 2022, from https://www.caranddriver.com/news/a39785992/elon-musk-tesla-robotaxi-2024/

11. Desk, H. (2022). Tesla’s robotaxi will be like Uber and Airbnb combined, hints Elon Musk. Retrieved 1 October 2022, from https://auto.hindustantimes.com/auto/electric-vehicles/teslas-robotaxi-will-be-like-uber-and-airbnb-combined-hints-elon-musk-41659927966965.html