If your scraper works perfectly for twenty requests and then starts returning login pages, empty carts or CAPTCHAs, the problem is usually not your parser. It is your session handling.
Most developers reach for rotating proxies first, on the assumption that a new IP on every request is the safest possible configuration. For a lot of targets, that assumption is wrong, and it is the direct cause of the failures they are trying to avoid.
Sticky sessions solve a specific class of problem. They also create a different one if you use them everywhere.
This article covers what a sticky session does at the protocol level, when it helps, when it hurts, and how to structure a job that needs both behaviors.
What a Sticky Session Actually Is
A sticky session, sometimes called a session-persistent proxy, keeps the same exit IP address for a defined period rather than assigning a new one per request.
That matters because HTTP itself is stateless. Everything that makes a website feel like a continuous experience is bolted on top, and most of it is bolted on with cookies.
According to MDN documentation, a session cookie is deleted when the client shuts down, while a persistent cookie expires at a time defined by the Expires or Max-Age attribute. Servers use both to reconstruct who you are across requests.
Cookie scope compounds this. Harvard's guidance on HTTP cookies notes that setting the domain attribute means the cookie is transmitted to that domain and all of its subdomains in every HTTP request.
Now consider what a server sees when you rotate IPs on every request while replaying the same cookie jar.
Request 1 arrives from Frankfurt with session cookie X. Request 2 arrives from São Paulo, three hundred milliseconds later, with the same session cookie X. Request 3 arrives from Manila.
No real user does that. The cookie says one person, the network says three continents, and the mismatch is trivially detectable.
How Residential Proxies Handle Session Length
This is where the provider configuration matters more than the scraper code.
SOAX exposes this behavior directly through its session controls, where residential proxies can either rotate the IP on every request or hold the same address for an extended period, with the session length set per task rather than fixed globally.
The mechanism is usually encoded in the proxy username itself, which is why proxy credential strings look so unusual. A typical request looks like this:
curl -x package-<id>-country-us-sessionid-number06-sessionlength-300:<pass>@proxy.soax.com:5000 \
-L https://example.com/product/12345Two parameters do the work. The sessionid identifies which session you are attaching to, and sessionlength defines how many seconds that IP stays assigned to it.
Change the sessionid and you get a different exit IP. Keep it identical and you get the same one until the session length expires.
That means your code controls rotation, not a black box. You can run twenty concurrent sessions, each with its own consistent identity, from a single set of credentials.
Where Sticky Sessions Are Required
Some workflows simply cannot function without IP consistency:
- Anything behind a login. The session cookie was issued to an IP. Changing it mid-session invalidates the login on many platforms.
- Multi-step flows. Search, then filter, then paginate, then open a detail page. Each step often depends on server-side state.
- Cart and checkout paths. Adding an item and then reading the cart from a different IP frequently returns an empty cart.
- Form submissions with CSRF tokens. The token is tied to the session that issued it.
- Any target that fingerprints network consistency alongside browser attributes.
If you are testing a checkout flow across ten locales, you want ten stable sessions, not one thousand random IPs.
Where Sticky Sessions Cause Problems
The reverse case is just as real, and it is what rotation exists for.
A sticky session concentrates all of your traffic through one address. If the target counts requests per IP, you will hit the ceiling faster and more visibly than if you had spread the load.
RFC 6585 defines the 429 status code as indicating that the user has sent too many requests in a given amount of time, and notes the response may include a Retry-After header. The same specification explicitly states that it does not define how the origin server identifies the user, and that the server might identify it by authentication credentials or by a stateful cookie.
That last clause is worth reading twice. Rate limiting is not always keyed to an IP address, which means rotating IPs does not automatically reset a counter.
For high-volume, stateless collection such as product listings, category pages or SERP snapshots, per-request rotation is the correct default. There is no state to preserve, and spreading load is the whole point.
A Practical Pattern
The realistic answer is not one or the other. It is choosing per task.
import requests
BASE = "package-{pkg}-country-{cc}"
def build_proxy(pkg, cc, session_id=None, length=300):
user = BASE.format(pkg=pkg, cc=cc)
if session_id:
user += f"-sessionid-{session_id}-sessionlength-{length}"
return {
"http": f"http://{user}:{PASSWORD}@proxy.soax.com:5000",
"https": f"http://{user}:{PASSWORD}@proxy.soax.com:5000",
}
# Stateless listing pages: rotate every request
proxies = build_proxy("1234", "us")
# Multi-step flow: pin one identity for five minutes
proxies = build_proxy(
"1234",
"us",
session_id="run-42",
length=300,
)Two rules keep this maintainable.
First, bind one requests.Session object to one proxy session. Sharing a cookie jar across rotating IPs recreates the exact mismatch described earlier.
Second, size the session length to the workflow, not to the maximum available. If a flow takes forty seconds, a sixty second session is enough. A sixty minute session simply gives the target more time to accumulate evidence about that IP.
Retry Logic Needs to Know Which Mode It Is In
This is the detail most scrapers get wrong.
When a stateless job fails, retrying on a fresh IP is correct. When a stateful job fails halfway through, retrying on a fresh IP is not a retry at all. It is a new session that has lost its cookies, its tokens and its cart.
Handle them differently:
- On a stateless 403 or 429, rotate the session ID and back off.
- On a stateful failure, restart the entire flow from step one with a new session ID.
- Respect Retry-After when the server sends it.
- Log which session ID produced which failure, so you can identify whether one exit IP is consistently blocked.
It is also worth remembering what you are permitted to collect in the first place. The Robots Exclusion Protocol was formally standardized in 2022, and the robots standard relies on voluntary compliance, which makes checking it a matter of professional practice rather than technical enforcement. Terms of service and applicable law still apply regardless of how your proxy is configured.
Rules of Thumb
Use rotating sessions for stateless, high-volume collection where every request is independent.
Use sticky sessions whenever server-side state exists, and pin the session for slightly longer than the flow needs.
Never share a cookie jar across rotating IPs.
Match session length to workflow duration rather than defaulting to the maximum.
Treat a mid-flow IP change as a full restart in your retry logic.
Sticky sessions are not a performance feature and not a stealth feature. They are a correctness feature, and treating them as optional is why a scraper that passed testing falls apart in production.
