Architecture of Information Systems
Search Engine

John Samuel
CPE Lyon

Year: 2017-2018
Email: john(dot)samuel(at)cpe(dot)fr

Outline: Search engine

Frontend development
Backend development
Application programming interface

Regular users
Domain Experts

Power users (Superusers)

Leonardo da Vinci (November 2018 Qwant results)

Leonardo da Vinci (October 2017 Google results)

Softwre (November 2018 Qwant Autocorrect)

Timeline

Map

Location of Archaelogical sites (Wikidata)

Need for Geographical Coordiantes

SELECT ?item ?location
WHERE {
  ?item wdt:P1435 wd:Q9259;
        wdt:P625 ?location
}

Queries

Keywords
Natural language queries

Search Results

Links
Snippets
Structured response
Natural language response

Search interface

One-box search
Advanced search (filters)

Personalized user experience

Filter search results (Multiple boxes)

Why filters?

Reduce information overload
Precise queries
Interactive search

Operators

AND
OR
NOT

Mnemonics

Time and location (Internationalization)

Past user search queries

Data collection
Data storage
Configuration
Logging
Dashboard
Security

Data ownership

Internal data (e.g., official website, internal wikis, databases etc.)
External data (e.g., other websites, wikis, open data)

Data model (Data and Schema)

Unstructured data (e.g., documents, texts, web pages etc.)
Semi-structured data (e.g., JSON/XML files etc.)
Structured data (e.g., relational databases, linked data)

Data sources

Web pages
Documents, texts
Sensors
Databases
...

Data acquisition

Data dumps
Crawlers
Web scraping
Application Programming Interface (API)

import requests
url = "https://api.github.com/"

response = requests.get(url)
print(response.json())

from SPARQLWrapper import SPARQLWrapper, JSON

sparql = SPARQLWrapper("http://query.wikidata.org/sparql")
sparql.setQuery("""
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q9143;
}
LIMIT 10
""")
sparql.setReturnFormat(JSON)
results = sparql.query().convert()

for result in results["results"]["bindings"]:
    print(result)

Data cleaning and transformation

Accuracy (e.g., verification with external sources)
Validity (e.g., detect constraint violations)
Uniformity (e.g., units)

Model
Indexation
Query optimization
Caching
Replication
Backup

Database schema
Schema-less

Relational Databases
Object-oriented Databases
NoSQL databases (e.g., graph databases)
NewSQL databases (SQL + ACID guarantees)

Document indices

Forward index
Inverted index

Database Indexation

Query Optimization

Join ordering
Cost estimation

Frequently asked questions and cached responses

Machines (servers, disks etc.)
Software packages and dependencies
Energy consumption

Development setup
Pre-production setup
Production setup

Containers (e.g., Linux containers, Docker)

Server-side
Client-side

A/B Testing

Access logs
Error logs
Event logs
Transaction logs

Why logs?

Debugging
Security (e.g., detect intrusion)
Database rollbacks
Audit
Analysis (e.g., detecting patterns, resource planning)

IP address
User
Resource ID
...

Wikimedia (Availability: 5^th October 2017)

Performance Metrics (active users, queries served etc.)
Real-time metrics (e.g., downtime, latency, throughput)

Email Alerts
Visual indicators

Data protection
Logged-in users or public access
Third party access

Authentication
Authorization to third party access

Intrusion
SQL code injection
Cross-site scripting
Denial of service

Service-oriented (SOAP)
Resource-oriented (REST)

XML
JSON
HTML?

Create
Read
Update
Delete
List

Complete data dumps
Selective data dumps

HTTP
Software development kits (SDKs)

Human-readable documentation
Machine-readable documentation (WSDL, WADL etc.)
Human and machine-readable documentation (microformats, semantic web languages)

Read documentation
Develop application to integrate
Add business logic, if any

Fully autonomous solution to integrate
Add business logic, if any

Resource usage limits

Limits on API call count (per user, IP)
Limits on data transfer
Temporary blocks

Analysis on frequently made API calls
Resource planning and allocation

No password
Basic authentication (e.g., username, password)
(Open) authentication protocols (e.g., OAuth)

Architecture of Information Systems Search Engine

Architecture of Information Systems

Outline: Search engine

1. Frontend development: Search Engine

1. Frontend development: Search Engine

1. Frontend development: Search Engine

1. Frontend development: Search Engine

1. Frontend development: Autocomplete

1. Frontend development: Search Engine

1. Frontend development: Search Engine Filters

1. Frontend development: Search Engine Filters

1. Frontend development: Search Engine Filters

1. Frontend Development: Target Audience

1. Frontend Development: Target Audience

1. Frontend development: Results

1. Frontend development: Results

1. Frontend development: Results

1. Frontend development: Autocorrect Results

1. Frontend development: Autocorrect Results

1. Frontend development: Results

Timeline

1. Frontend development: Results

Map

1. Frontend development: Results

Need for Geographical Coordiantes

1. Frontend development: Results

Queries

Search Results

1. Front-end development

Search interface

Personalized user experience

1. Frontend development: Advanced search (filters)

Filter search results (Multiple boxes)

1. Frontend development: Advanced search (filters)

1. Frontend development: Advanced search (filters)

1. Frontend development: Advanced search (filters)

Why filters?

1. Frontend development: Advanced Search in one-box

Operators

1. Frontend development: Advanced Search in one-box

Mnemonics

1. Frontend development: Personalized user experience

Time and location (Internationalization)

1. Frontend development: Personalized user experience

Past user search queries

2. Backend development

2.1 Backend development: Data collection

Data ownership

Data model (Data and Schema)

2.1 Backend development: Data collection

Data sources

2.1 Backend development: Data collection

Data acquisition

2.1 Backend development: Data collection

2.1 Backend development: Data collection

2.1 Backend development: Data collection

2.1 Backend development: Data collection

2.1 Backend development: Data collection

2.1 Backend development: Data collection

Data cleaning and transformation

2.2 Backend development: Data storage

2.2. Backend development: Data Model

2.2 Backend development: Data storage

2.2. Document indices and Query Optimization

Document indices

Database Indexation

Query Optimization

2.2. Backend development: Caching

2.2. Backend development: Replication and Backup

2.3. Resource management and configuration

2.3. Resource management and configuration

2.3. Deployment

2.3. Packaging

2.3. Load balancing

2.3. Selective Testing

A/B Testing

2.4. Backend development: Logging

2.4. Logging

Why logs?

2.4. Logging

Architecture of Information Systems
Search Engine