Architecture of Information Systems
Search Engine

John Samuel
CPE Lyon

Year: 2017-2018
Email: john(dot)samuel(at)cpe(dot)fr

Outline: Search engine

• Frontend development
• Backend development
• Application programming interface

• Regular users
• Domain Experts

Search interface

• One-box search
• Advanced search (filters)

Personalized user experience

Queries

• Keywords
• Natural language queries

Search Results

• Links
• Structured response
• Natural language response

Filter search results (Multiple boxes)

Why filters?

• Reduce information overload
• Precise queries
• Interactive search

Operators

• AND
• OR
• NOT

Mnemonics

Time and location (Internationalization)

Past user search queries

• Data collection
• Data storage
• Configuration
• Logging
• Dashboard
• Security

Data ownership

• Internal data (e.g., official website, internal wikis, databases etc.)
• External data (e.g., other websites, wikis, open data)

Data model (Data and Schema)

• Unstructured data (e.g., documents, texts, web pages etc.)
• Semi-structured data (e.g., JSON/XML files etc.)
• Structured data (e.g., relational databases, linked data)

Data sources

• Web pages
• Documents, texts
• Sensors
• Databases
• ...

Data acquisition

• Data dumps
• Crawlers
• Web scraping
• Application Programming Interface (API)

Data cleaning and transformation

• Accuracy (e.g., verification with external sources)
• Validity (e.g., detect constraint violations)
• Uniformity (e.g., units)

• Model
• Indexation
• Query optimization
• Caching
• Replication
• Backup

• Database schema
• Schema-less

• Relational Databases
• Object-oriented Databases
• NoSQL databases (e.g., graph databases)
• NewSQL databases (SQL + ACID guarantees)

Document indices

• Forward index
• Inverted index

Database Indexation

Query Optimization

• Join ordering
• Cost estimation

• Frequently asked questions and cached responses

• Machines (servers, disks etc.)
• Software packages and dependencies
• Energy consumption

• Development setup
• Pre-production setup
• Production setup

• Containers (e.g., Linux containers)

• Server-side
• Client-side

A/B Testing

• Access logs
• Error logs
• Event logs
• Transaction logs

Why logs?

• Debugging
• Security (e.g., detect intrusion)
• Database rollbacks
• Audit
• Analysis (e.g., detecting patterns, resource planning)

• IP address
• User
• Resource ID
• ...

• Performance Metrics (active users, queries served etc.)
• Real-time metrics (e.g., downtime, latency, throughput)

• Email Alerts
• Visual indicators

• Data protection
• Logged-in users or public access
• Third party access

• Authentication
• Authorization to third party access

• Intrusion
• SQL code injection
• Cross-site scripting
• Denial of service

• Service-oriented (SOAP)
• Resource-oriented (REST)

• XML
• JSON

• Create
• Read
• Update
• Delete
• List

• Complete data dumps
• Selective data dumps

• HTTP
• Software development kits (SDKs)

• Human-readable documentation
• Machine-readable documentation (WSDL, WADL etc.)
• Human and machine-readable documentation (microformats, semantic web languages)

1. Read documentation
2. Develop application to integrate
3. Add business logic, if any

1. Fully autonomous solution to integrate
2. Add business logic, if any

Resource usage limits

• Limits on API call count (per user, IP)
• Limits on data transfer
• Temporary blocks

• Analysis on frequently made API calls
• Resource planning and allocation

• No password
• Basic authentication (e.g., username, password)
• (Open) authentication protocols (e.g., OAuth)

References

• https://en.wikipedia.org/wiki/Information_system
• https://en.wikipedia.org/wiki/Search_engine_(computing)
• https://query.wikidata.org/
• https://weather.com/
• https://duckduckgo.com/bang
• http://commoncrawl.org/
• https://en.wikipedia.org/wiki/High_availability
• https://grafana.wikimedia.org
• https://status.wikimedia.org
• http://highscalability.com/

Image credits

• Wikimedia Commons
• http://histropedia.com/timeline/5bnj79bjyr/Artists
• https://pixabay.com/