Django website help!

[snoopy]

Alight all, I've made some good progress on my project, and I've hit a wall.

I have it rendering a schedule:



Using the following bit of code:

Code: Select all

.....    # Create the listings data.
    chandata = []
    for channel in schedule.models.Channel.objects.order_by('alias'):
        chan = {'name':channel.alias, 'icon':channel.icon}
        programs = []
        for program in channel.program_set.filter(
                           stop__gt=schstart + tablestep).filter(
                           start__lt=schstop).order_by('start'):
            programs.append(
                {'progobj':program, 'colspan':program.colspan(
                    tablestep, schstart, schstop)})
            #TODO Need to do some schedule overlap checking and stuff.....
            #TODO this assumes that they are just back to back

        chan['programs'] = programs
        chandata.append(chan)

    datadict['chandata'] = chandata

    return render_to_response('schedule/index.html', datadict)

So, here's what I want to add... and have yet to figure out a good, efficient way to pull it off:

I want to add both filter and exclude abilities to the table, while maintaining the basic table structure.

So, Imagine that I click on a "filter" link for "The Voice" - from there I want the server to render a new table, showing only "The Voice" shows. Now, if I just stuck with the same time frame and channels it'd be pretty easy... but also pretty useless.

So, when the table renders, I want the following to happen: I want the time scale to be able to break up into time portions that matter, I want the programs to compress, and I want channels with no matches to drop out... all in a decently clean and cheap algorithm, generating something like this:



I'm having trouble figuring a good way to code this up. I guess maybe I just have to live with it being fairly expensive... every time I come at it, I end up in a place where I need to cycle over the array a couple times; first building up a table that's bigger than it needs to be, then compressing it down and truncating it to a reasonable length.

Anyone have any brilliant ideas of how I should approach it?

[snoopy]

Here's how I'm thinking of approaching it:

Start with all programs that end later than the schedule start time
Apply the filter list to it
Apply the exclude list to it

Create a list of all of the channels that still apply

Run a while loop: Start at schedule start time, step forward in time by the increment:

If no program is running at the time and we haven't had any yet, just bump the schedule start time by an increment

If a program is running:

If it's new, add it to the list to render, at a length of one unit, in a row corresponding to its channel
If it's the same as what was found last block, bump the length by one unit but don't create a new block
If it's the same as what what found in the last block for all channels, don't bump the length & mark the time block as skipped


If no program is running, but we've had ones in the past:

If it's the first block after a program, add an empty block
If it's the continuation of a blank spot, and another channel has a program running, extend the empty block
If it's the continuation of a blank spot for all channels, don't extend the block, but flag the times as being skipped


In a basic form, I think this will do what I want. It's kinda ugly in that it's got a whole big load of conditional statements.... I wish I could think of a cleaner way to do it but right now I've got nothin'.

[Jeff250]

Can you fix the pictures so that they are large enough to see?

[snoopy]

Done. Thought I had linked it. Apparently not.

[Jeff250]

First, congratulations on the great progress.

If I understand you, it seems like there are at least two problems that we can flesh out here. One is the problem of elegant and efficient inclusion and exclusion filters. The second is the problem of trying to elegantly and efficiently render the compressed table after you have the desired shows.

The phpBB, for instance, tackles the search problem by (if I recall correctly) having a separate table for keywords with a many-to-many relation between posts and keywords. So a search is just a database query. And, although it doesn't look like phpBB has this feature, excluding by keywords should be just making the database query a little longer. One problem with this approach is that it is unforgiving to partial word matches, but that might not be a problem. Without doing something like this, I don't think you can improve much upon taking multiple iterations yourself over the input data. Databases have the LIKE and NOT LIKE operators, which would basically be the database doing this for you. Keep in mind that that might not be so bad though, depending on the size of the array. Your show titles are much shorter on average than DBB posts. It might work just fine in practice.

For rendering the compressed table, this might be made easier with an interval tree to keep track of the compressed time intervals:
http://en.wikipedia.org/wiki/Interval_tree
Here's a python implementation that I've used for a small project that seemed to work well enough:
http://pypi.python.org/pypi/interval/1.0.0

[Jeff250]

P.S.: I almost forgot: Some databases like MySQL have "full text search," which is the many-to-many relation that I described earlier except implemented directly by the database itself. It has similar tradeoffs, and it's probably a bit quicker than setting up the relation yourself, but I don't think you can use this through django's ORM though.

[snoopy]

Thanks! It's been a fun project. Some days I wish I didn't have a job and school to go to so I could work on it more.... darn responsibilities.

For the filter/exclusion/search features (I intend to have all three in place eventually.):

Django has a "full text search" filter built into the ORM that I intend to use: https://docs.djangoproject.com/en/dev/r ... ts/#search

For the filter and exclusion filters: I intend to only allow them to be applied to title and subtitles - and I purposely want them to be an exact match filter. The intention is that the user won't generate the filter field, but will just have a link to click on that will be associated with a particular program.... so when I click on a filter for the title of "The Voice" I don't want matches that are close... I only want The Voice. I'll have the search feature in there for user generated filter fields.

I got the bulk of the logic written.... I think you're right that the interval tree is right along the lines of something that I could have used...

I plan to get some testing/cleaning up of the code in tonight, then I'll post it up.

[fliptw]

where are you getting the channel information from?

[snoopy]

http://www.schedulesdirect.org/

It's a subscription service, and can't be used with for-profit programs.

I pull the data using xmltv scripts. Once the project gets past the "home project" state, I'll contact them about having my program added to their list of approved programs. Right now it qualifies because it's a personal project....

Apparently.... sadly.... it's loosing subscribers....

[Jeff250]

Django has a "full text search" filter built into the ORM

Cool, I wasn't aware they wrapped that. They usually err on the side of database agnosticism, but it's nice to see them support this.

Some days I wish I didn't have a job and school to go to so I could work on it more....

Hacker pipe dream.

[snoopy]

It isn't running completely yet, but right now it's super slow. I need to see if fixing a bit of function speeds it up but I doubt it. So, I'm going to need to reevaluate to trim things down. The problem is that it cycles through the majority of the database.

[Jeff250]

This is useful for debugging database query bottlenecks:
http://djangosnippets.org/snippets/93/

And this for python bottlenecks:
https://code.djangoproject.com/wiki/ProfilingDjango
(From the link above, I've used the runprofileserver.py script, but you should see which you prefer.)

[snoopy]

Nice... thanks for the tip....

11851 queries... I'm thinking that I'm doing something wrong here....

[Jeff250]

https://docs.djangoproject.com/en/dev/r ... ct-related

It's better to use the form where you specify the attributes manually.

Ideally, the # of queries per page shouldn't grow as the # of things on the page do.

[snoopy]

Nice.... I'm re-factoring & giving up a little bit of what I was hoping to do...

Anyways, I've hit a puzzle.... I can't quite figure out what I'm doing wrong (I'm actually starting to wonder if it's a bug?)

This snippet cycles through all the program hits, and builds up a dictionary with data for the template to render... mostly it works great.

Code: Select all

        for p in programs:
            # If we haven't already encountered this channel, add it.
            if not p.channel in chandata:
                 chandata[p.channel] = {
                     'alias':p.channel.alias,
                     'icon':p.channel.icon,
                     'programs':[]
                     }

            # Adjust the start and stop times if the program starts
            # before the schedule or ends after.
            if p.start < schstart:
                start = 0
            else:
                start = (p.start-schstart).seconds/tablestep.seconds

            if p.stop > schstart + displaydur:
                stop = displaydur.seconds/tablestep.seconds
            else:
                stop = (p.stop-schstart).seconds/tablestep.seconds

            chandata[p.channel]['programs'].append(
                {
                    'start':start,
                    'stop':stop,
                    'span':stop-start,
                    'obj':p,
                    #'reclink':#TODO,
                    #'favlink':#TODO,
                    #'locklink':TODO,
                    'titlefilter': filurl(
                        webroot, baseurl, fl, el,
                        newf = ('sltitle', p.sltitle)),
                    'titleexcl': filurl(
                        webroot, baseurl, fl, el,
                        newe = ('sltitle', p.sltitle)),
                    'subfilter': filurl(
                        webroot, baseurl, fl, el,
                        newf = ('slsub_title', p.slsub_title)),
                    'subexcl': filurl(
                        webroot, baseurl, fl, el,
                        newe = ('slsub_title', p.slsub_title)),
                })

.....

def filurl(webroot, basedict, fl, el, newf = False, newe = False):
    '''Generate an encoded url that will contain the link
       for a new filter or exclude.'''

    # Parse the filter list into a string.
    if newf:
        fl[newf[0]]=newf[1]
        basedict['filter'] = fl

    # Parse the exclude list into a string.
    if newe:
        el[newe[0]]=newe[1]
        basedict['exclude'] = el

    # Get the overall url.
    return webroot + '?' + urllib.urlencode(basedict)

Here's the part that has a problem: theoretically, I create four url's for each program - the fulurl function takes a base set of get parameters (those passed when the page view is called), modifies the one little pertinent spot, and kicks out an encoded link.

So.. assuming I load the page with no GET parameters, it should generate links with just one get parameter - the one pertaining to the particular link. Well, it's generating links with "leftover" parameters from previous iterations...

Do you see anything that I'm doing wrong, or did I find some sort of weird bug? I'm sure it's something that I'm doing wrong that isn't hitting me....

[snoopy]

I got it solved by creating a copy of basedict, fl, and el at the beginning of the function via the dict.copy() method.

I guess it's something weird about django? Normally wouldn't each call of the function start with a fresh set of variables... not ones left over from the last time?

[Jeff250]

Objects in python are references, so when you say y = x, or pass x to a function, you're just copying the reference to the object, not the object itself.

So if we have
x -> [dict]
after y = x, we have
x -> [dict]
y ------^
(both x and y pointing to the same dict).

I'm wondering if using reverse URL resolution and setting up the URL's in your urls.py file wouldn't be simpler than using urllib?

[snoopy]

Objects in python are references, so when you say y = x, or pass x to a function, you're just copying the reference to the object, not the object itself.

So if we have
x -> [dict]
after y = x, we have
x -> [dict]
y ------^
(both x and y pointing to the same dict).

I'm wondering if using reverse URL resolution and setting up the URL's in your urls.py file wouldn't be simpler than using urllib?

Yes... but.

if I have:

Code: Select all

def foo(dict, p, bar):
    dict[p] = bar
    print dict

and then I call it a bunch of times:

Code: Select all

key = 10
diffdict = {}

for i in range(0,10):
    foo(diffdict, key, key-5)
    key += 1

hrm.... I get:

Code: Select all

{10: 5}
{10: 5, 11: 6}
{10: 5, 11: 6, 12: 7}
{10: 5, 11: 6, 12: 7, 13: 8}
{10: 5, 11: 6, 12: 7, 13: 8, 14: 9}
{10: 5, 11: 6, 12: 7, 13: 8, 14: 9, 15: 10}
{10: 5, 11: 6, 12: 7, 13: 8, 14: 9, 15: 10, 16: 11}
{10: 5, 11: 6, 12: 7, 13: 8, 14: 9, 15: 10, 16: 11, 17: 12}
{10: 5, 11: 6, 12: 7, 13: 8, 14: 9, 15: 10, 16: 11, 17: 12, 18: 13}
{10: 5, 11: 6, 12: 7, 13: 8, 14: 9, 15: 10, 16: 11, 17: 12, 18: 13, 19: 14}

out.... interesting. I guess the same applies to all mutable objects?

I thought that each function had its own namespace (is that the right term?)
So, if I called a function and gave it a dictionary, the dictionary from the calling function and the dictionary in the called function would be two different things.

This may explain a few other minor bugs I've identified in my code.

[snoopy]

I'll also take a look at the url dispatcher.

[Jeff250]

Inside a function,

x = y

modifies the local namespace,

but

somedict[x] = y

modifies that dict, not the local namespace. somedict itself will be looked up in the local namespace (or so on) but anyone who has a reference to it will be able to see changes you make to somedict.

What if you were instead modifying a list? Would your intuition still be that modifications to that list shouldn't be visible outside that function?

[fliptw]

the word you are looking for is scope - basically from where something can be accessed, and how long its accessible.

references cross scope boundaries.

[snoopy]

Inside a function,

x = y

modifies the local namespace,

but

somedict[x] = y

modifies that dict, not the local namespace. somedict itself will be looked up in the local namespace (or so on) but anyone who has a reference to it will be able to see changes you make to somedict.

What if you were instead modifying a list? Would your intuition still be that modifications to that list shouldn't be visible outside that function?

My intuition is that the same should apply to all variables....

That's why I referenced mutable objects vs. non-mutable ones. For mutable objects, I now understand that a reference is passed, and the object is changed. For non-mutable objects, assignment (in general) doesn't modify the existing object, it creates a new reference to a new one.

I guess I'm saying that I was missing part of an important difference between mutable and non-mutable objects; in what happens during different methods of assignment.

[snoopy]

Alright... puzzle me this, batman!

Code: Select all

el = {}

    if 'exclude' in get:
        #supertest = el
        el = ast.literal_eval(get['exclude'])
        supertest = el
        #supertest = ast.literal_eval(get['exclude'])

when I print "supertest" on my page:

for the first "supertest = el", before reassigning el: supertest = {}
for the second "supertest = el", after reassigning el: supertest = {sltitle': [u'reporters', u'the-price-is-right', u'talk-philly', .........]} (Wrong!)
for the third "supertest = ast.literal_eval(get['exclude'])", directly assigning the same thing to supertest and what I just finished assigning to el: supertest = {'sltitle': [u'reporters']} (Right!)

What's going on? Why isn't el getting assigned to correctly?

[Jeff250]

I haven't figured out what you're doing with literal_eval yet, but one note is that if el is global and your code fragment is from inside a function, saying

el = bletch

will create a new local variable named el. You need to say something like globals()['el'] = bletch to make explicit your intent. Note that lookups are different: if there is no el in local scope, then it's assumed you mean the global one, but python can't use this reasoning for assignment, where it's unclear if you want to create a new local variable or assign to an old global one. Another reason why globals can be evil.

edit: If this isn't it, can you post a small self-contained example?

[snoopy]

yeah, never mind... I got it.

It's another nuance/twist to it that I wasn't thinking about that got me.

I'm building up a structure like this:

datadict = {'filters':{'category of field':

• }}
  
  So, I was creating a copy of the dictionary, but not the list.... and wasn't realizing that the copy thing was recursive to everything inside of it...
  
  So, that problem wasn't really anywhere close to there....
  
  Here's what I had to fix:
  

  Code: Select all

  def filurl(webroot, basedict, newf = False, newe = False):
      '''Generate an encoded url that will contain the link
         for a new filter or exclude.'''
  
      newd = dict(basedict)
  
      # Parse a new filter.
      if newf:
          if 'filter' in newd and newf[0] in newd['filter']:
              newfil = newd['filter'].copy()
              newlist = list(newd['filter'][newf[0]])
              newlist.append(newf[1])
              newfil[newf[0]] = newlist
              newd['filter'] = newfil
          else:
              if not 'filter' in newd:
                  newd['filter']={}
  
              newfil = newd['filter'].copy()
              newfil[newf[0]] = [newf[1]]
              newd['filter'] = newfil
  
      # Parse a new exclude.
      if newe:
          if 'exclude' in newd and newe[0] in newd['exclude']:
              newfil = newd['exclude'].copy()
              newlist = list(newd['exclude'][newe[0]])
              newlist.append(newe[1])
              newfil[newe[0]] = newlist
              newd['exclude'] = newfil
          else:
              if not 'exclude' in newd:
                  newd['exclude']={}
  
              newfil = newd['exclude'].copy()
              newfil[newe[0]] = [newe[1]]
              newd['exclude'] = newfil
  
      # Get the overall url.
      return webroot + '?' + urllib.urlencode(newd)
  
  

  So, I had to create copies of all of the mutable objects in the structure (I guess duh)