graphic cues superimposed on video sequences

Hi, we are considering to buy Direct RT and I would like to know if it allows the presentation of visual cues (e.g. gif images) superimposed to short video sequences.

If this is not the case, is the solution proposed below correct?

If graphic cues couldn't be superimposed to video sequences these cues could be inserted in the video sequences (a rather tedious task) by editing single frames. Since I am interested in recording reaction times to these signals, should I keep the video frame rate the same as the monitor refresh rate? Please, keep in mind that I'm interested in measuring the reaction time to these graphic cues (participants would have to press a button when the cues appear on the screen). As far as I understand, if video frame rate and monitor refresh rate are the same, I should be able to get the reaction time to the graphic cues by substracting the reaction time mesured from the beginning of video presentation minus the cue presentation starting time.

Would this approach work?

Thanks in advance,

Alvaro

Hi Alvarom,

Awesome question. First off I think we *may* have added a function to superimpose graphics over video but I will need to check. For now, let's assume not and I'll try to answer your other question. In theory, absolutely, yes--if you have a 100Hz display which refreshes every 10ms, then a 10ms refresh rate would match nicely with a video encoded with 10ms frames. That sounds beautiful but in world of real things--I don't expect it would work that precisely. First, your encoder software would have to know you are really actually very serious about the 10ms frame rate--I don't know how much variability there is in that field. The other important variable I can not tell you is whether or not the frame rate would be matched perfectly with the screen refresh rate. In the vast majority of cases (I'm *guessing*) the frame rate would not be equal to the refresh rate (or equal to multiples of each other). Consequently, I can't see anybody having put effort into syncing them in the rare cases where they could be precisely coordinated.

So all of that is to say superimposing text over the video, or elsewhere on the screen (which you *certainly* can do) would be more accurate.

One thought--if there is any video codec out there that allows you to execute programmable events within the video you could take advantage of that. I have some recollection that .wmv (?) files many years ago would let you trigger events during the video. If you could embed text *and* send a TTL signal at the same time THEN you could know exactly when that text was presented.

Anyways, those are some initial thoughts. I will check through the code though and see if we got anywhere with text overlays in v2010. I wouldn't hold my breath but I do recall the effort to make it happen.

Looks like you can not overlay a text or image stimulus upon a video--even if you use a clear value of 0 for the text. The video will simply overwrite the space (the text or image will actually be behind it). *However* I tried playing a second video over top of the initial video after 1000ms delay. It had no problem paying on top. So if you could create your overlay stimulus/stimuli in any kind of video format (mpg, avi, animated gif) then you could locate and display it exactly as you like with as much precision as possible.

There was only significant problem I couldn't immediately work around. Videos do not automatically clear themselves from the screen. So the stimulus remained. Maybe this could be cured to some extent with an animated gif (or simply clearing the second video to a blank frame when it finishes) but you would still not see that region of the underlying video again. Some way to completely clear the second video without affecting the first would be necessary for this to be an easy method.

Some initial thoughts!

Hi Jarvis,

Thanks a lot for your replies. I tried to do that -I've read in a different blog that the problem most likely arose for the text or graphic cues being still images. Unfortunatelly, when the videos were supposed to be shown, the experiment froze completely. Probably this was due to my lack of experience with Direct RT :)

I'll try to get both videos playing (most likely video and animated gif) and then try to find solution for clearing the 'cue video'.

I may come back to you in a couple of days.

Best,

Alvaro

Hi again, Jarvis,

I still couldn't get to play the second video on top of the first one, but I did manage to play an animated gif on top of a video. However, I still faced some problems.

I thought that the best way to avoid the problem of the top video not clearing was to use an animated gif with a transparent frame showing for a long time. However, transparent areas/frames of animated gifs appear solid black. These areas appear transparent in Mozilla Firefox, MS PowerPoint and Open Office Impress.

Just for the record, there is an image distortion everytime a new loop starts.

I still couldn't manage to get a second video playing on top of a first one. When I try to do that, DRT freezes and get the following error message: "Empirisoft executeDRT has detected a problem and needs to close". At this point, my screen resolution changes from 1280x800 to 800x600.

Still, the most critical point would be to get rid of the black rectangle where the animated gif (or eventually the video) was showing.

Any ideas on how to proceed?

Best,

Alvaro

What version of DirectRT (20xx.x.x.x) are you running and on what version of Windows?

I'm using version 2010.2.103.1115 on Windows XP.

Alvaro

DirectRT v2010 has a new feature where you can launch an executable (.exe) file. You use ^ as the prefix and copy the exe into your stim folder (or perhaps supply the full path if it's elsewhere). E.g., if your exe were called [I]popup.exe[/I], you would simply enter ^popup in the STIM column.

Haven't experimented with it much yet but this morning I did try to launch an exe after a video and it was more than happy to appear on top of the movie as long as I used a "0" as the clear value in the LOC column. You can use a 0ms delay in the time column after the movie begins or enter an onset time such as "1000" to delay the start of the exe from when the movie begins playing.

I think this may be a solution. All that is needed is a short exe that simply presents whatever you like and is timed to disappear after a fixed length of time.

Here is an example. I've attached an exe file that flashes some text on the screen with a clear background. In this case it flashes for about 250ms (could be made more precise and flexible, but keep in mind it's just for an initial demonstration).

You can unzip it and place it in your stim folder as text.exe. Then in you input file, you simply request ^text in the STIM column (since that the name of the exe file). If you put a delay of say 1000ms after the movie starts, you will see this text appear and quickly disappear without a trace. Just be sure to use *,*,0 as the LOC value for the exe so it won't clear the video file from the screen.

Note that the text could be replaced with anything--an image, a video, an animated gif, a website, or whatever.

Hi Jarvis,

Since I have no experience in programming started to investigate how to create a exe file. The solution I came with was to create a bat file and then convert it to an exe one. However I faced some problems. If I call a program for playing a video a frame still shows (the program skin). Your solution seems to be the answer. Do you think you could post some basic instructions on how to create the exe file? (Not sure if this is the place to ask such a thing, but would be really important for me to be able to create those files myself, and most likely this could help other researchers).

Thanks in advance!

Alvaro

Here are two samples two start you off. They both display a stimulus with a clear background for 250ms. [I]eatPopcorn[/I] is written in Visual Basic 6.0, and [I]beHappy[/I] is written in Visual Basic Express 2010. Many people still have VB6 kicking around and like it so there's that. VB Express is actually free from Microsoft and the programming for a task like this is surprisingly easy. In the latter, you'll see there is simply a form with an image (a "smiley face" and a text label "be happy!". Then if you click on the form you'll see its properties. I set a bunch of them to do this but the critical one is "set transparency color" (or something like that). I made the background green ("lime" actually) and defined that as the transparent color. Added a timer that starts automatically and closes the form after 250ms. That's pretty much it. The VB6 version is very similar--you would just open "clearBackground.vbp" as the project file.

If you download VB Express, you would simply open the beHappy.sln file, make edits and then request VB to "build the solution". This will produce the exe file.

Let me know if you have any questions!

[I]Note to you programmer types (you know who you are!):[/I] These programs [I]of course[/I] could also be made WAY more efficient by using dynamic stimuli rather than having fixed text or images. But again, this makes for a nice simple demo I think. And a non-programmer could make separate exe's for various stimuli without getting into anything more complex--just change the text, add a picture, or other stimulus and re-build the exe under a new name. If that's the case, it probably wouldn't take the use more than 5-20 minutes to make any desired changes.

Finally, technical thingee: For the VB Express version ([I]beHappy[/I]), the .NET 4.0 redistributable package is required to be installed. This can be obtained from [URL]http://download.microsoft.com[/URL]. VB Express will probably install it automatically, but you may need to install it on other machines running your experiment.

All feedback, comments, suggestions, improvements welcome! Feel free to post improved or modifed versions with some explanation if you think other users would like to see them in action!

Hi,

It's been quite a long time since my last post. In the last couple of weeks, we've started working again on this somehow forgotten project. I'd like to present the graphic cue on top of the video at random times, and measure the reaction time to this graphic signal (the subject would have to press a key).

The solution proposed by jarvis24 seemed very elegant (see above). On a few test experiments I got the desired graphic cue flashing over the clip. However the image presented by the exe file seemed to start much later than expected.

When trying different conditions, times, etc, I faced some problems which I couldn't solve. In some additional tests, I programmed the .exe file to start 10000 ms after the video has started (Video Stim - Loc: 0,01; Time: 10000; Exe Stim - Loc: 0,0,0; Time: rt:any) and had the following problems:
[LIST=1][*]If I press any key after the exe file has been executed, the video closes down (the video clip is 2 min long).[*]If I don't press any keys, the video ends 10 seconds after the start. If I want to use random delays, I would have to use random times for the exe stimulus and the video would, most likely, end at those times.[*]In the log file, it's not really clear when the second stimuli (exe file) has started.[*]Are reaction times accurate when using exe files as stimuli?[/LIST]Any ideas on how to overcome these problems?

Thanks in advance!

alvarom

If I insert the graphic cues in the video clip itself, could I get a reaction time in relation to the beginning of the clip and still have the video play to the end?

This could be quite tedious, but -if possible- might be a way to circumvent the limitations...

Best,

alvarom

After posting the last message I realized that we had already discussed about the possibility of inserting the graphic cues on the video itself. Since other solutions are not so straight forward, I guess I'll try check how accurately the video is reproduced. Would it make sense to record the experiment session (using some appropriate software), and then to analyze the frames of this recording? If I do this for a couple of video clips, would I be on the safe side?

Thanks again...

alvarom