问题 使用线程时Django中的数据库错误


我正在使用需要查询PostgreSQL数据库的Django Web应用程序。使用Python实现并发时 穿线 界面,我得到了 DoesNotExist 查询项目的错误。当然,顺序执行查询时不会发生这些错误。

让我展示一个单元测试,我写这个测试来演示意外的行为:

class ThreadingTest(TestCase):
    fixtures = ['demo_city',]

    def test_sequential_requests(self):
        """
        A very simple request to database, made sequentially.

        A fixture for the cities has been loaded above. It is supposed to be
        six cities in the testing database now. We will made a request for
        each one of the cities sequentially.
        """
        for number in range(1, 7):
            c = City.objects.get(pk=number)
            self.assertEqual(c.pk, number)

    def test_threaded_requests(self):
        """
        Now, to test the threaded behavior, we will spawn a thread for
        retrieving each city from the database.
        """

        threads = []
        cities = []

        def do_requests(number):
            cities.append(City.objects.get(pk=number))

        [threads.append(threading.Thread(target=do_requests, args=(n,))) for n in range(1, 7)]

        [t.start() for t in threads]
        [t.join() for t in threads]

        self.assertNotEqual(cities, [])

如您所见,第一个测试按顺序执行一些数据库请求,这确实没有问题。但是,第二个测试执行完全相同的请求,但每个请求都是在一个线程中生成的。这实际上是失败的,返回一个 DoesNotExist 例外。

执行此单元测试的输出如下:

test_sequential_requests (cesta.core.tests.threadbase.ThreadingTest) ... ok
test_threaded_requests (cesta.core.tests.threadbase.ThreadingTest) ...

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/jose/Work/cesta/trunk/src/cesta/core/tests/threadbase.py", line 45, in do_requests
    cities.append(City.objects.get(pk=number))
  File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/manager.py", line 132, in get
    return self.get_query_set().get(*args, **kwargs)
  File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/query.py", line 349, in get
    % self.model._meta.object_name)
DoesNotExist: City matching query does not exist.

...其他线程返回类似的输出...

Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/jose/Work/cesta/trunk/src/cesta/core/tests/threadbase.py", line 45, in do_requests
    cities.append(City.objects.get(pk=number))
  File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/manager.py", line 132, in get
    return self.get_query_set().get(*args, **kwargs)
  File "/home/jose/Work/cesta/trunk/parts/django/django/db/models/query.py", line 349, in get
    % self.model._meta.object_name)
DoesNotExist: City matching query does not exist.


FAIL

======================================================================
FAIL: test_threaded_requests (cesta.core.tests.threadbase.ThreadingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jose/Work/cesta/trunk/src/cesta/core/tests/threadbase.py", line 52, in test_threaded_requests
    self.assertNotEqual(cities, [])
AssertionError: [] == []

----------------------------------------------------------------------
Ran 2 tests in 0.278s

FAILED (failures=1)
Destroying test database for alias 'default' ('test_cesta')...

请记住,所有这些都发生在PostgreSQL数据库中,该数据库应该是线程安全的,而不是SQLite或类似物。测试也使用PostgreSQL运行。

在这一点上,我完全迷失了可能失败的东西。有什么想法或建议吗?

谢谢!

编辑: 我写了一个小视图,只是为了检查它是否在测试中运行。以下是视图的代码:

def get_cities(request):
    queue = Queue.Queue()

    def get_async_cities(q, n):
        city = City.objects.get(pk=n)
        q.put(city)

    threads = [threading.Thread(target=get_async_cities, args=(queue, number)) for number in range(1, 5)]

    [t.start() for t in threads]
    [t.join() for t in threads]

    cities = list()

    while not queue.empty():
        cities.append(queue.get())

    return render_to_response('async/cities.html', {'cities': cities},
        context_instance=RequestContext(request))

请不要考虑在视图代码中编写应用程序逻辑的愚蠢行为。请记住,这只是一个概念证明,并不会永远不会在真正的应用程序中。

结果是代码运行良好,请求在线程中成功完成,视图最终在调用其URL后显示城市。

因此,我认为使用线程进行查询只会在您需要测试代码时出现问题。在生产中,它将毫无问题地工作。

有没有成功测试这种代码的有用建议?


7888
2018-06-08 12:06


起源

你确定夹具是进口的吗?你可以粘贴说:“Fixture demo_city处理”或类似的东西......啊,没关系..只是没有读完整的问题.. - Tisho


答案:


尝试使用TransactionTestCase:

class ThreadingTest(TransactionTestCase):

TestCase将数据保存在内存中,不会向数据库发出COMMIT。可能是线程正在尝试直接连接到DB,而数据尚未提交到那里。在这里看到描述: https://docs.djangoproject.com/en/dev/topics/testing/?from=olddocs#django.test.TransactionTestCase

除了方式之外,TransactionTestCase和TestCase是相同的   其中数据库重置为已知状态和能力   测试代码以测试提交和回滚的效果。一个   TransactionTestCase在测试运行之前重置数据库   截断所有表并重新加载初始数据。一个   TransactionTestCase可以调用commit和rollback并观察   这些调用对数据库的影响。


12
2018-06-08 13:16



这是方法 - Jonatas CD


这听起来像是交易的问题。如果您正在当前请求(或测试)中创建元素,那么它们几乎肯定是在未提交的事务中,无法从另一个线程中的单独连接访问。你可能需要 手动管理您的转化 让这个工作。


2
2018-06-08 12:09



实际上,查询不是修改任何数据,而只是从数据库中检索它。所以,我没有在你的注释中得到重点。无论如何,我已经改变了请求 do_requests 函数由原始SQL查询和结果相同。 - José L. Patiño
docs.djangoproject.com/en/1.8/topics/db/transactions 答案中的链接不再起作用,因为它适用于v1.4。 - HostedMetrics.com


从这部分文档中变得更加清晰

class LiveServerTestCase(TransactionTestCase):
    """
    ...
    Note that it inherits from TransactionTestCase instead of TestCase because
    the threads do not share the same transactions (unless if using in-memory
    sqlite) and each thread needs to commit all their transactions so that the
    other thread can see the changes.
    """

现在,事务尚未在TestCase中提交,因此其他线程看不到更改。


0
2018-02-22 07:13