A face search engine is four well-documented components in a row: a face detector, an aligner, an embedding network and a nearest-neighbour index, all fed by a crawler. Each stage has a parameter that decides who gets matched and who gets missed. Every one of those parameters is also a privacy decision, so it pays to know where they sit before writing code.
Detection and alignment
The detector finds each face and a handful of landmarks. RetinaFace (Deng and colleagues, 2019) is one open option. It is a single-stage detector trained with five manually annotated facial landmarks on the WIDER FACE dataset, it scored 91.4% average precision on that dataset's hard test set, and its authors report real-time speed on a single CPU core for a VGA image when using a light backbone. The landmarks drive alignment. A similarity transform maps them to fixed template positions, so every face reaches the embedding network at the same scale and rotation.
This stage is also where to reject bad input. NIST's 2019 identification report lists the images that break real databases: files with no face, several people in one frame, files that cannot be decoded, photos rotated by 90 or 180 degrees, and faces printed on clothing. A query with zero faces or more than one should return an error asking for a clearer photo before any GPU time is spent.
Embedding: a face becomes a vector
FaceNet (Schroff, Kalenichenko and Philbin, 2015) trained a network to map faces into a compact Euclidean space where distance corresponds to face similarity. It learned from triplets of matching and non-matching faces, used 128 bytes per face, and reported 99.63% on the LFW benchmark. ArcFace changed the training objective to an additive angular margin on the softmax loss to maximise separation between identities. In practice you normalise each vector to unit length and compare by cosine similarity.
One line in the ArcFace abstract should change how you store these vectors. The authors show that the pre-trained model "can generate identity-preserved face images" from feature vectors, using only the network gradient and batch-normalisation statistics. An embedding can be turned back into a recognisable face, so the vector table needs the same controls as the photos: encryption at rest, access logging, and deletion whenever the source image is deleted.
The index: approximate nearest neighbour search
For a small collection, brute force is fine. FAISS's IndexFlatIP does exact inner-product search, and its documentation notes it works "also for cosine (normalize vectors beforehand)". At larger scales you start trading recall for speed.
An IVF index partitions the space into nlist cells with k-means and scans only nprobe of them per query. The FAISS wiki puts the fraction of the database compared at roughly nprobe/nlist and names the failure mode, which is missing the true nearest neighbour when its cell is not one of those probed. HNSW, from Malkov and Yashunin, builds a layered proximity graph instead. M sets the number of neighbours per node, while efConstruction and efSearch set how deep the graph is explored at build time and at query time.
One limitation should shape the whole design. The FAISS wiki states that "HNSW does not support removing vectors from the index", because removal "would destroy the graph structure". Flat indexes do support remove, but it shrinks the index and renumbers the ids. If people can ask to be taken out of your results, and they should be able to, you need a deny-list checked on every query plus a scheduled rebuild that drops tombstoned ids for good.
A query path with those pieces in place looks like this:
AGE_GATE = 21 # set above 18 to absorb age-estimation error
def search(image, index, deny_ids, threshold, k=50):
faces = detect(image) # boxes plus five landmarks
if len(faces) == 0:
return reject("no face found, try a clearer photo")
if len(faces) > 1:
return reject("upload a photo with exactly one face")
crop = align(image, faces[0].landmarks)
if estimated_age(crop) < AGE_GATE:
return reject("searches of minors are not allowed")
v = normalize(embed(crop)) # unit length: inner product = cosine
scores, ids = index.search(v, k) # approximate top-k
return [(s, source_url(i)) for s, i in zip(scores, ids) if i not in deny_ids and s >= threshold] # pages, never names
Age estimation is imperfect, which is why NIST evaluates it in its own track, so treat the gate as one control among several and keep the margin generous.
Crawling public images
The crawler decides what the index can ever return. RFC 9309, the IETF standard for robots.txt published in September 2022, asks crawlers to honour its rules and says they SHOULD NOT rely on a cached copy for more than 24 hours. It also says the rules "are not a form of access authorization", and an allowed path says nothing about whether the people in the photos agreed to anything.
Store the page URL, the image URL and the crawl date with every vector. Results can then link to a page the user can inspect, and a takedown can be traced to every vector that came from it.
The law draws a harder edge. Article 5(1)(e) of the EU AI Act prohibits AI systems "that create or expand facial recognition databases through the untargeted scraping of facial images from the internet or CCTV footage", and the Act's prohibitions have applied since 2 February 2025. "Untargeted" is the word your crawler's scope will be judged against, and that is a question for a lawyer before the first fetch.
Thresholds and what the user sees
The index returns the k nearest vectors for every query, so it always returns something. NIST's FRTE 1:N evaluation calls this investigation mode: a threshold of zero, a fixed list of 50 candidates, and an assumption that "a human will be used to review the candidates". Set a cosine threshold tuned on impostor pairs drawn from images like your production traffic, and show only what clears it.
FaceSearch's explainer on how facial recognition works puts the point plainly for non-engineers: a match is a distance below a threshold, and the threshold is a choice. That is also the honest description of what a face search returns. It shows public pages where a similar face appears, sometimes under another name. It cannot see private accounts or messages, and an empty list only means nothing cleared the threshold on the pages that were indexed.
Guardrails to ship in version one
- Adults only. Require account holders to be 18 or over, refuse queries the age gate flags, and put the ban on searching minors in the terms.
- Purpose and consent. Before each search, ask the user to confirm a lawful reason and their right to use the image. If you enrol faces of Illinois residents, BIPA requires written notice and a written release before collection.
- Removal that works. Accept requests by source URL and by face, add matches to the deny-list at once, and drop them at the next rebuild.
- Short retention. Delete uploaded images and query embeddings when the search completes. BIPA also requires a public, written retention schedule.
- Abuse logging. Record account, time, model version, threshold and result count, leave the image out, and alert on accounts that search many unrelated faces.
- Results as links. Return source pages and scores, and never print a name the system inferred.
Write the deletion job before the crawler, then add a test that indexes a face, files a removal request and asserts that the face is absent from the next query.
