Data Science for Chemists

IPL Summer School, CPE Lyon

3. Data Analysis and visualization

John Samuel
CPE Lyon

Year: 2023-2024
Email: john.samuel@cpe.fr

Data acquisition

1. Surveys
• Manual surveys
• Online surveys
2. Sensors¹
• Temperature, pressure, humidity, rainfall
• Acoustic, navigation
• Proximity, presence sensors
3. Social networks
4. Video surveillance cameras
5. Web

1. https://en.wikipedia.org/wiki/List_of_sensors

Data storage formats

• Binary and Textual Files
• CSV/TSV
• XML
• JSON
• Media (Images/Audio/Video)

Types of data stores

1. Structured data stores
   • Relational databases
   • Object-oriented databases
2. Unstructured data stores
   • Filesystems
   • Content-management systems
   • Document collections
3. Semi-structured data stores
   • Filesystems
   • NoSQL data stores

ACID Transactions¹

• Atomicity: Each transaction must be "all or nothing".
• Consistency: Any transaction must bring database from one valid state to another.
• Isolation: Both concurrent execution and sequential execution of transactions must bring the database to same state.
• Durability: Irrespective of power losses, crashes, a transaction once committed to the database must remain in that state.

1. https://en.wikipedia.org/wiki/ACID

ACID Transactions

• Ensure validity of databases even in case of errors, power failures
• Important in banking sector

Types of data stores

• Relational databases
• Object-oriented databases
• NoSQL (Not only SQL) data stores
• NewSQL

NoSQL

• Comprises consistency
• Focus on availability and speed

Types of NoSQL stores

• Column-oriented database
• Document-oriented database
• Key-value database
• Graph-oriented database

Data extraction techniques

• Data dumps
  • Downloading complete data dumps
  • Downloading selective data dumps
• Periodical polling of data feeds (e.g., blogs, news feeds)
• Data streams
  • Subscrbing to data streams (push notifications)

Query interfaces

• Query endpoints supporting declarative languages
  • SQL
  • SPARQL
• Automated Manual search (and filter) options

3.3. Crawlers for web pages

Application Programming Interface (API)

• Web operations (CRUD) to manipulate externally managed resources
• Requires programmers to develop wrappers for web service integration

Data Cleaning: Types of Errors

• Syntactical errors
• Semantical errors
• Data coverage errors

Syntactical errors

• Lexical errors (e.g., user entered a string instead of a number)
• Data format errors (e.g, order of last name, first name)
• Irregular data errors (e.g., usage of different metrics)

Semantic errors

• Violation of integrity constraints
• Contradiction
• Duplication
• Invalid data (unable to detect despite presence of triggers and integrity constraints)

Coverage errors

• Missing values
• Missing data

Handling Syntactical errors

• Validation using schema (e.g., XSD, JSONP)
• Data transformation

Handling Semantic errors

• Duplicate elimination using techniques like specifying integrity constraints like functional dependencies

Handling Coverage errors

• Interpolation techniques
• External data sources

Administrators and handling errors

• User feedback
• Alerts and triggers

Languages

• Template languages
• XSLT
• AWK
• Sed
• Programming languages like PERL

ETL (Extraction Transformation and Loading)

1. Data Extraction
2. Data Cleaning
3. Data Transformation
4. Loading data to information stores

Models for data analysis

• Multidimensional data analysis
• Dimensions
  • Attributes
  • Levels
  • Hierarchies
• Facts
  • Measures

Models for data analysis

• Multidimensional data analysis: Examples
• Dimensions (e.g.Spatio-temporal dimensions, Product)
  • Attributes (e.g. Name, Manufactures etc.)
  • Levels (e.g., Day, Month, Quarter, Store, City, Country etc.)
  • Hierarchies (e.g., Day-Month-Quarter-Year, Store-City-Country etc.)
• Facts
  • Measures (e.g., Number of products sold/unsold)

Star Schema

Data Cubes

• Data cubes for online analytical processing (OLAP)
• OLAP Cube operations
  • Slice
  • Dice
  • Drill up/down
  • Pivot

Snow Schema

ETL: From one data store to another

• From: Data sources
  • Internal or external databases
  • Web Services
• To: Data warehouses
  • Enterprise warehouses
  • Web warehouses

Activities of data analysis

1. Retrieving values
2. Filter
3. Compute derived values
4. Find extremum
5. Sort
6. Determine range
7. Characterize distribution
8. Find analysis
9. Cluster
10. Correlate
11. Contextualization

1. https://en.wikipedia.org/wiki/Data_analysis

Data Visualization

1. Time-series
2. Ranking
3. Part-to-whole
4. Deviation
5. Sort
6. Frequency distribution
7. Correlation
8. Nominal comparison
9. Geographic or geospatial

1. https://en.wikipedia.org/wiki/Data_visualization

Data Visualization: Examples

1. Bar-chart (Nominal comparison)
2. Pie-chart (part-to-whole)
3. Histograms (frequency-distribution)
4. Scatter-plot (correlation)
5. Network
6. Line-chart (time-series)
7. Treemap
8. Gantt chart
9. Heatmap

Pie Chart

k Predominant colours

RGB Scatter plots (Comparison)

Sites web

• https://jupyter.org/
• https://www.wikidata.org/

Couleurs

• Color Tool - Material Design

Images

• Wikimedia Commons