filesystem blobstore does not atomically replace objects #490
Description
Hello, I am using the latest s3proxy image with the filesystem backend in our development environment.
In our use case we have concurrent access to blobs but when we modify blobs we see that s3proxy report "NoSuchKey when calling the GetObject operation" eventually.
This issue is related to the old reported one in https://issues.apache.org/jira/browse/JCLOUDS-835 that seems to be solved years ago, but seems to fail the s3 commitment "a mutating operation like write or overwrite should succeed and expose the new object or fail and retain the old object."
Is there any explanation to the filesystem backend doesn't implement atomic behavior?
The issue can be reproduced with this code:
$ cat qa-reader.py
import boto3
import os
import socket
import time
import urllib3
urllib3.disable_warnings()
s3_url = os.environ['s3_url']
pid = os.getpid()
hostname = socket.gethostname()
client = boto3.client(
's3',
endpoint_url=s3_url,
aws_access_key_id="demouser",
aws_secret_access_key="demouser123",
verify=False
)
while True:
obj = client.get_object(Bucket='qa-test', Key='qa-file.txt')
msg = obj['Body'].read().decode('utf-8')
print(msg,end='\r')
$ cat qa-writer.py
import boto3
import os
import socket
import time
s3_url = os.environ['s3_url']
pid = os.getpid()
hostname = socket.gethostname()
client = boto3.client(
's3',
endpoint_url=s3_url,
aws_access_key_id="demouser",
aws_secret_access_key="demouser123",
verify=False
)
while True:
msg = f"Updated by {hostname} (pid={pid})"
client.put_object(Bucket='qa-test', Key='qa-file.txt', Body=str(msg))
print("File updated +",end='\r')
time.sleep(200/1000)
print("File updated #",end='\r')
$ s3_url='http://127.0.0.1:9102' python qa-writer.py &
$ s3_url='http://127.0.0.1:9102' python qa-reader.py
Traceback (most recent call last):69605)
File "qa-reader.py", line 19, in <module>
obj = client.get_object(Bucket='qa-test', Key='qa-file.txt')
File "/home/k8sadmin/s3-qatest/.venv/lib/python3.8/site-packages/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/k8sadmin/s3-qatest/.venv/lib/python3.8/site-packages/botocore/client.py", line 960, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
Thanks in advance!